2024-04-24T00:20:23.902 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T00:20:23.902 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533 --description rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_deep_copy} --verbose -- /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/orig.config.yaml 2024-04-24T00:20:23.935 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T00:20:24.019 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533 branch: squid description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_deep_copy} email: ceph-qa@ceph.com first_in_suite: false job_id: '7670533' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9514 sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 sleep_before_teardown: 0 subset: 69/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIM1uFhWanLvyIzTiPYVwYfZyZvMwYROwt4gjDdoui72/apTLiOuZUn4tfs/iRVmS2c3b1yqfEiPs0RUrelx7yE= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPMmKtpw4oM6/fecT9g5lapXKGYUK7S9AQfIwQYxCtfbqmnCmgSfN5b+su8ADMcyUnCsfH0VtgDhTDW4L+7Rn44= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 deep_copy: true ops: 6000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T00:20:24.019 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa; will attempt to use it 2024-04-24T00:20:24.020 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa/tasks 2024-04-24T00:20:24.020 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T00:20:24.021 INFO:teuthology.task.internal:Checking packages... 2024-04-24T00:20:24.045 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b98aab6c00f483888c049d0d8e7bc507e115dd86' 2024-04-24T00:20:24.045 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T00:20:24.045 INFO:teuthology.packaging:ref: None 2024-04-24T00:20:24.045 INFO:teuthology.packaging:tag: None 2024-04-24T00:20:24.045 INFO:teuthology.packaging:branch: squid 2024-04-24T00:20:24.045 INFO:teuthology.packaging:sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:20:24.046 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-24T00:20:24.253 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:20:24.255 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T00:20:24.261 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T00:20:24.262 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T00:20:24.276 INFO:teuthology.task.internal:Saving configuration 2024-04-24T00:20:24.288 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T00:20:24.294 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T00:20:24.319 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 00:14:29.988601', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICV6xNfZYV1GT5gqIsLk13vjFygg5e48W7x5HXMStMgU'} 2024-04-24T00:20:24.344 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 00:14:29.987039', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFEdir+NhGvojqRCSoaDC8jEeQK8Tbl6kybAxof0X+bg'} 2024-04-24T00:20:24.344 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T00:20:24.351 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-04-24T00:20:24.351 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-24T00:20:24.351 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T00:20:24.426 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb778ace280>, signals=[15]) 2024-04-24T00:20:24.427 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T00:20:24.433 INFO:teuthology.task.internal:Opening connections... 2024-04-24T00:20:24.434 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:20:24.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:20:24.515 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:20:24.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:20:24.598 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T00:20:24.605 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-04-24T00:20:24.611 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-04-24T00:20:24.612 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-04-24T00:20:24.659 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-24T00:20:24.659 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="Ubuntu" 2024-04-24T00:20:24.659 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="22.04" 2024-04-24T00:20:24.659 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-24T00:20:24.659 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_CODENAME=jammy 2024-04-24T00:20:24.659 INFO:teuthology.orchestra.run.smithi064.stdout:ID=ubuntu 2024-04-24T00:20:24.660 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE=debian 2024-04-24T00:20:24.660 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T00:20:24.660 INFO:teuthology.orchestra.run.smithi064.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T00:20:24.660 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T00:20:24.660 INFO:teuthology.orchestra.run.smithi064.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T00:20:24.660 INFO:teuthology.orchestra.run.smithi064.stdout:UBUNTU_CODENAME=jammy 2024-04-24T00:20:24.660 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-04-24T00:20:24.680 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-04-24T00:20:24.685 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-04-24T00:20:24.685 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="Ubuntu" 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="22.04" 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_CODENAME=jammy 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:ID=ubuntu 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE=debian 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T00:20:24.733 INFO:teuthology.orchestra.run.smithi167.stdout:UBUNTU_CODENAME=jammy 2024-04-24T00:20:24.734 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-04-24T00:20:24.755 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T00:20:24.965 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T00:20:24.971 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T00:20:24.972 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-04-24T00:20:24.974 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-04-24T00:20:24.979 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T00:20:24.985 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T00:20:24.986 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-04-24T00:20:25.020 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-04-24T00:20:25.027 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T00:20:25.087 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T00:20:25.101 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T00:20:25.102 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T00:20:25.102 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T00:20:25.103 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T00:20:25.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:20:25.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:20:25.108 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-04-24T00:20:25.108 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-04-24T00:20:25.153 INFO:teuthology.orchestra.run.smithi167.stdout:5.15.0-56-generic 2024-04-24T00:20:25.154 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.15.0-56-generic 2024-04-24T00:20:25.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get clean 2024-04-24T00:20:25.155 INFO:teuthology.orchestra.run.smithi064.stdout:5.15.0-56-generic 2024-04-24T00:20:25.156 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.15.0-56-generic 2024-04-24T00:20:25.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2024-04-24T00:20:25.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2024-04-24T00:20:25.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2024-04-24T00:20:25.442 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:20:25.450 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T00:20:25.461 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T00:20:25.461 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T00:20:25.537 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:20:25.539 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-24T00:20:25.619 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T00:20:25.660 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-24T00:20:25.701 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,392 kB] 2024-04-24T00:20:25.735 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-24T00:20:25.742 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,392 kB] 2024-04-24T00:20:25.791 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-24T00:20:25.796 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-24T00:20:25.797 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-24T00:20:25.797 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,768 kB] 2024-04-24T00:20:25.825 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-24T00:20:25.833 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-24T00:20:25.839 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T00:20:25.839 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-24T00:20:25.846 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-24T00:20:25.858 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-24T00:20:25.861 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-24T00:20:25.863 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-24T00:20:25.863 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-24T00:20:25.864 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,768 kB] 2024-04-24T00:20:25.885 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-24T00:20:25.887 INFO:teuthology.orchestra.run.smithi167.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-24T00:20:25.887 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-24T00:20:25.888 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-24T00:20:25.888 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-24T00:20:25.898 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-24T00:20:25.901 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T00:20:25.901 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-24T00:20:25.906 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-24T00:20:25.907 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-24T00:20:25.913 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-24T00:20:25.913 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-24T00:20:25.919 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-24T00:20:25.932 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-24T00:20:25.934 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-24T00:20:25.955 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-24T00:20:25.957 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-24T00:20:25.961 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-24T00:20:25.962 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-24T00:20:25.962 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-24T00:20:25.964 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T00:20:25.964 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-24T00:20:25.974 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-24T00:20:25.980 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-24T00:20:25.982 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-24T00:20:25.995 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-24T00:20:25.999 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-24T00:20:25.999 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-24T00:20:26.000 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-24T00:20:26.034 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-24T00:20:26.035 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-24T00:20:26.035 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-24T00:20:26.036 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-24T00:20:26.037 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-24T00:20:26.082 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-24T00:20:26.083 INFO:teuthology.orchestra.run.smithi167.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-24T00:20:26.083 INFO:teuthology.orchestra.run.smithi167.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-24T00:20:26.084 INFO:teuthology.orchestra.run.smithi167.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-24T00:20:26.084 INFO:teuthology.orchestra.run.smithi167.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-24T00:20:26.120 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-24T00:20:26.429 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-24T00:20:26.470 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-24T00:20:26.485 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-24T00:20:26.486 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-24T00:20:26.549 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-24T00:20:26.550 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-24T00:20:26.557 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T00:20:26.721 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-24T00:20:27.354 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-24T00:20:27.391 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-24T00:20:27.404 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-24T00:20:27.405 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-24T00:20:27.405 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-24T00:20:27.407 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-24T00:20:27.408 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-24T00:20:27.408 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-24T00:20:27.436 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-24T00:20:27.438 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-24T00:20:27.603 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-24T00:20:27.686 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-24T00:20:27.770 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-24T00:20:27.851 INFO:teuthology.orchestra.run.smithi064.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-24T00:20:27.878 INFO:teuthology.orchestra.run.smithi064.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-24T00:20:31.706 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 13.2 MB in 2s (7,357 kB/s) 2024-04-24T00:20:32.397 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 13.2 MB in 3s (5,236 kB/s) 2024-04-24T00:20:32.986 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-04-24T00:20:33.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T00:20:33.061 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-04-24T00:20:33.257 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2024-04-24T00:20:33.258 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2024-04-24T00:20:33.420 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:20:33.420 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:20:33.420 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:20:33.421 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:20:33.421 INFO:teuthology.orchestra.run.smithi167.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-24T00:20:33.421 INFO:teuthology.orchestra.run.smithi167.stdout: python2.7-minimal 2024-04-24T00:20:33.421 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:20:33.422 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2024-04-24T00:20:33.422 INFO:teuthology.orchestra.run.smithi167.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T00:20:33.422 INFO:teuthology.orchestra.run.smithi167.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-24T00:20:33.423 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-24T00:20:33.424 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2024-04-24T00:20:33.424 INFO:teuthology.orchestra.run.smithi167.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-24T00:20:33.424 INFO:teuthology.orchestra.run.smithi167.stdout:Recommended packages: 2024-04-24T00:20:33.424 INFO:teuthology.orchestra.run.smithi167.stdout: thermald 2024-04-24T00:20:33.476 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2024-04-24T00:20:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T00:20:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-24T00:20:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-24T00:20:33.478 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages will be upgraded: 2024-04-24T00:20:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T00:20:33.535 INFO:teuthology.orchestra.run.smithi167.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T00:20:33.535 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 113 MB of archives. 2024-04-24T00:20:33.535 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-24T00:20:33.535 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-24T00:20:33.770 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-04-24T00:20:33.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T00:20:33.839 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-04-24T00:20:34.023 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2024-04-24T00:20:34.024 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2024-04-24T00:20:34.035 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-24T00:20:34.166 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-24T00:20:34.185 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:20:34.185 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:20:34.185 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:20:34.185 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:20:34.186 INFO:teuthology.orchestra.run.smithi064.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-24T00:20:34.186 INFO:teuthology.orchestra.run.smithi064.stdout: python2.7-minimal 2024-04-24T00:20:34.186 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:20:34.187 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2024-04-24T00:20:34.187 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T00:20:34.187 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-24T00:20:34.187 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-24T00:20:34.188 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2024-04-24T00:20:34.188 INFO:teuthology.orchestra.run.smithi064.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-24T00:20:34.188 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2024-04-24T00:20:34.188 INFO:teuthology.orchestra.run.smithi064.stdout: thermald 2024-04-24T00:20:34.240 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2024-04-24T00:20:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T00:20:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-24T00:20:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-24T00:20:34.242 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2024-04-24T00:20:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T00:20:34.325 INFO:teuthology.orchestra.run.smithi064.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T00:20:34.325 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 113 MB of archives. 2024-04-24T00:20:34.325 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-24T00:20:34.325 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-24T00:20:35.180 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-24T00:20:35.371 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-24T00:20:35.507 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-24T00:20:35.508 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-24T00:20:35.508 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-24T00:20:35.744 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-24T00:20:35.802 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-24T00:20:36.184 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 113 MB in 2s (48.9 MB/s) 2024-04-24T00:20:36.358 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-24T00:20:36.496 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-24T00:20:36.496 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-24T00:20:36.497 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-24T00:20:36.638 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-24T00:20:36.660 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-24T00:20:37.053 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 113 MB in 2s (47.0 MB/s) 2024-04-24T00:20:37.231 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-24T00:20:38.390 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-24T00:20:38.395 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:20:38.481 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:39.167 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-24T00:20:39.171 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:20:39.261 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:40.625 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-24T00:20:40.647 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:20:40.725 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:41.162 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-24T00:20:41.182 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:20:41.220 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:41.645 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-24T00:20:41.666 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:20:41.743 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:42.306 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-24T00:20:42.327 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:20:42.364 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:47.981 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:20:48.108 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:20:48.448 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:20:48.562 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:20:48.700 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:20:48.825 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:20:48.856 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-24T00:20:48.866 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-24T00:20:48.906 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T00:20:49.125 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:20:49.228 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:20:49.506 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-24T00:20:49.529 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-24T00:20:49.573 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T00:20:54.949 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-24T00:20:54.986 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:20:55.023 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:55.244 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-24T00:20:55.282 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T00:20:55.319 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:57.541 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:20:57.663 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:20:57.996 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T00:20:58.072 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T00:20:58.118 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:58.198 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T00:20:58.236 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-24T00:20:58.339 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:58.549 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T00:20:58.670 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:58.796 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-24T00:20:58.924 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:20:59.650 INFO:teuthology.orchestra.run.smithi167.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:20:59.651 INFO:teuthology.orchestra.run.smithi167.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-24T00:20:59.651 INFO:teuthology.orchestra.run.smithi167.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:20:59.651 INFO:teuthology.orchestra.run.smithi167.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-24T00:20:59.817 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:21:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:21:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-24T00:21:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:21:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-24T00:21:00.426 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:21:01.628 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:21:02.304 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:21:03.028 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-24T00:21:03.129 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-24T00:21:03.239 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:21:03.308 INFO:teuthology.orchestra.run.smithi167.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T00:21:03.308 INFO:teuthology.orchestra.run.smithi167.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:21:03.713 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-24T00:21:03.839 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-24T00:21:03.965 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T00:21:04.014 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T00:21:04.014 INFO:teuthology.orchestra.run.smithi064.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:21:18.002 INFO:teuthology.orchestra.run.smithi167.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T00:21:18.002 INFO:teuthology.orchestra.run.smithi167.stdout:Sourcing file `/etc/default/grub' 2024-04-24T00:21:18.020 INFO:teuthology.orchestra.run.smithi167.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T00:21:18.039 INFO:teuthology.orchestra.run.smithi167.stdout:Generating grub configuration file ... 2024-04-24T00:21:18.243 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T00:21:18.243 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub' 2024-04-24T00:21:18.259 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T00:21:18.276 INFO:teuthology.orchestra.run.smithi064.stdout:Generating grub configuration file ... 2024-04-24T00:21:18.344 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:21:18.362 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:21:18.501 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:21:18.515 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:21:18.717 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:21:18.725 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T00:21:18.791 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T00:21:18.797 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T00:21:18.820 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:21:18.825 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T00:21:18.890 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T00:21:18.895 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T00:21:18.977 INFO:teuthology.orchestra.run.smithi167.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T00:21:18.977 INFO:teuthology.orchestra.run.smithi167.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T00:21:18.977 INFO:teuthology.orchestra.run.smithi167.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T00:21:19.031 INFO:teuthology.orchestra.run.smithi064.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T00:21:19.032 INFO:teuthology.orchestra.run.smithi064.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T00:21:19.032 INFO:teuthology.orchestra.run.smithi064.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T00:21:19.055 INFO:teuthology.orchestra.run.smithi167.stdout:done 2024-04-24T00:21:19.070 INFO:teuthology.orchestra.run.smithi064.stdout:done 2024-04-24T00:21:19.444 DEBUG:teuthology.orchestra.run.smithi167:> dpkg -s linux-image-generic 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Package: linux-image-generic 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Status: install ok installed 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Priority: optional 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Section: kernel 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Installed-Size: 21 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: amd64 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Source: linux-meta 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Version: 5.15.0.105.102 2024-04-24T00:21:19.473 INFO:teuthology.orchestra.run.smithi167.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T00:21:19.474 INFO:teuthology.orchestra.run.smithi167.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T00:21:19.474 INFO:teuthology.orchestra.run.smithi167.stdout:Recommends: thermald 2024-04-24T00:21:19.474 INFO:teuthology.orchestra.run.smithi167.stdout:Description: Generic Linux kernel image 2024-04-24T00:21:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: This package will always depend on the latest generic kernel image 2024-04-24T00:21:19.474 INFO:teuthology.orchestra.run.smithi167.stdout: available. 2024-04-24T00:21:19.474 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-24T00:21:19.474 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T00:21:19.474 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T00:21:19.475 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-24T00:21:19.475 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T00:21:19.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get clean 2024-04-24T00:21:19.543 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2024-04-24T00:21:19.603 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 21 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.15.0.105.102 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2024-04-24T00:21:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2024-04-24T00:21:19.605 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-24T00:21:19.605 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T00:21:19.605 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T00:21:19.605 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-24T00:21:19.605 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T00:21:19.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2024-04-24T00:21:19.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2024-04-24T00:21:19.739 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T00:21:19.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2024-04-24T00:21:19.854 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:21:19.935 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T00:21:20.061 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:21:20.061 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T00:21:20.114 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T00:21:20.195 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T00:21:20.289 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T00:21:20.489 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 119 kB in 1s (165 kB/s) 2024-04-24T00:21:20.516 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 110 kB in 1s (198 kB/s) 2024-04-24T00:21:21.475 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-04-24T00:21:21.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T00:21:21.508 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-04-24T00:21:21.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T00:21:21.547 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-04-24T00:21:21.592 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-04-24T00:21:21.726 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2024-04-24T00:21:21.726 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2024-04-24T00:21:21.794 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2024-04-24T00:21:21.795 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2024-04-24T00:21:21.885 INFO:teuthology.orchestra.run.smithi167.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-24T00:21:21.885 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:21:21.886 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:21:21.886 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:21:21.886 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:21:21.886 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T00:21:21.886 INFO:teuthology.orchestra.run.smithi167.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T00:21:21.886 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:21:21.925 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T00:21:21.927 DEBUG:teuthology.orchestra.run.smithi167:> dpkg -s linux-image-generic 2024-04-24T00:21:21.968 INFO:teuthology.orchestra.run.smithi064.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-24T00:21:21.968 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:21:21.968 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:21:21.968 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:21:21.969 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:21:21.969 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T00:21:21.970 INFO:teuthology.orchestra.run.smithi064.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T00:21:21.970 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Package: linux-image-generic 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Status: install ok installed 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Priority: optional 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Section: kernel 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Installed-Size: 21 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: amd64 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Source: linux-meta 2024-04-24T00:21:21.984 INFO:teuthology.orchestra.run.smithi167.stdout:Version: 5.15.0.105.102 2024-04-24T00:21:21.985 INFO:teuthology.orchestra.run.smithi167.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T00:21:21.985 INFO:teuthology.orchestra.run.smithi167.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T00:21:21.985 INFO:teuthology.orchestra.run.smithi167.stdout:Recommends: thermald 2024-04-24T00:21:21.985 INFO:teuthology.orchestra.run.smithi167.stdout:Description: Generic Linux kernel image 2024-04-24T00:21:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: This package will always depend on the latest generic kernel image 2024-04-24T00:21:21.985 INFO:teuthology.orchestra.run.smithi167.stdout: available. 2024-04-24T00:21:21.985 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-04-24T00:21:22.011 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T00:21:22.012 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2024-04-24T00:21:22.028 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2024-04-24T00:21:22.028 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2024-04-24T00:21:22.028 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2024-04-24T00:21:22.028 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2024-04-24T00:21:22.028 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 21 2024-04-24T00:21:22.028 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T00:21:22.028 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2024-04-24T00:21:22.029 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2024-04-24T00:21:22.029 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.15.0.105.102 2024-04-24T00:21:22.029 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T00:21:22.029 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T00:21:22.029 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2024-04-24T00:21:22.029 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2024-04-24T00:21:22.029 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2024-04-24T00:21:22.029 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2024-04-24T00:21:22.029 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-04-24T00:21:22.030 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.VGZqownfXt 2024-04-24T00:21:22.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub/grub.cfg /tmp/tmp.VGZqownfXt 2024-04-24T00:21:22.074 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.RFJmYhomVR 2024-04-24T00:21:22.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub/grub.cfg /tmp/tmp.RFJmYhomVR 2024-04-24T00:21:22.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.VGZqownfXt 2024-04-24T00:21:22.130 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.RFJmYhomVR 2024-04-24T00:21:22.202 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.VGZqownfXt is 10KB 2024-04-24T00:21:22.216 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.VGZqownfXt 2024-04-24T00:21:22.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T00:21:22.248 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.RFJmYhomVR is 10KB 2024-04-24T00:21:22.261 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.RFJmYhomVR 2024-04-24T00:21:22.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T00:21:22.276 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:21:22.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T00:21:22.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T00:21:22.322 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:21:22.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T00:21:22.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T00:21:22.345 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-24T00:21:22.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo update-grub 2024-04-24T00:21:22.390 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-24T00:21:22.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo update-grub 2024-04-24T00:21:23.538 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub' 2024-04-24T00:21:23.540 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T00:21:23.541 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-04-24T00:21:23.639 INFO:teuthology.orchestra.run.smithi167.stderr:Sourcing file `/etc/default/grub' 2024-04-24T00:21:23.640 INFO:teuthology.orchestra.run.smithi167.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T00:21:23.642 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2024-04-24T00:21:23.701 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:21:23.706 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:21:23.878 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T00:21:23.884 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T00:21:23.922 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:21:23.928 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T00:21:23.965 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T00:21:23.969 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T00:21:24.074 INFO:teuthology.orchestra.run.smithi064.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T00:21:24.074 INFO:teuthology.orchestra.run.smithi064.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T00:21:24.074 INFO:teuthology.orchestra.run.smithi064.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T00:21:24.092 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-04-24T00:21:24.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-04-24T00:21:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T00:21:24.168 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T00:21:24.208 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T00:21:24.213 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T00:21:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T00:21:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T00:21:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T00:21:24.341 INFO:teuthology.orchestra.run.smithi167.stderr:done 2024-04-24T00:21:24.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2024-04-24T00:21:54.132 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-24T00:21:54.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-24T00:21:54.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:21:54.353 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-24T00:21:54.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-24T00:21:54.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:22:12.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-04-24T00:22:12.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-24T00:22:21.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-24T00:22:21.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:22:21.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-24T00:22:21.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:22:24.578 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-04-24T00:22:24.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-24T00:22:36.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-24T00:22:36.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:22:36.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-24T00:22:36.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:22:55.007 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-04-24T00:22:55.266 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-24T00:23:10.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-24T00:23:10.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:23:10.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-24T00:23:10.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:23:10.467 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-04-24T00:23:11.295 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-04-24T00:23:11.511 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-24T00:23:11.511 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-24T00:23:11.512 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-04-24T00:23:11.560 INFO:teuthology.orchestra.run.smithi064.stdout:5.15.0-105-generic 2024-04-24T00:23:11.560 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-24T00:23:11.560 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T00:23:11.560 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T00:23:12.561 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T00:23:12.561 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T00:23:12.776 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-04-24T00:23:12.820 DEBUG:teuthology.parallel:result is None 2024-04-24T00:23:29.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-24T00:23:29.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:23:29.536 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-04-24T00:23:30.076 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-24T00:23:30.076 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-24T00:23:30.076 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-04-24T00:23:30.123 INFO:teuthology.orchestra.run.smithi167.stdout:5.15.0-105-generic 2024-04-24T00:23:30.124 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-24T00:23:30.124 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T00:23:30.124 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T00:23:31.125 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-24T00:23:31.125 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T00:23:31.489 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2024-04-24T00:23:31.529 DEBUG:teuthology.parallel:result is None 2024-04-24T00:23:31.530 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T00:23:31.539 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T00:23:31.540 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T00:23:31.543 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T00:23:31.548 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T00:23:31.556 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T00:23:31.565 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T00:23:31.566 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T00:23:31.592 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T00:23:31.617 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T00:23:31.624 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T00:23:31.624 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T00:23:31.648 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T00:23:31.664 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T00:23:31.672 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T00:23:31.678 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T00:23:31.686 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T00:23:31.688 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T00:23:31.700 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T00:23:31.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T00:23:31.720 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T00:23:31.744 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T00:23:31.753 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T00:23:31.753 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T00:23:31.776 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T00:23:31.792 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T00:23:31.828 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T00:23:31.875 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:23:31.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T00:23:31.968 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T00:23:31.974 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T00:23:32.020 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:23:32.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T00:23:32.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-04-24T00:23:32.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-04-24T00:23:32.159 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T00:23:32.167 INFO:teuthology.task.internal:Starting timer... 2024-04-24T00:23:32.167 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T00:23:32.177 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T00:23:32.185 DEBUG:teuthology.task.selinux:Excluding smithi064: OS 'ubuntu' does not support SELinux 2024-04-24T00:23:32.185 DEBUG:teuthology.task.selinux:Excluding smithi167: OS 'ubuntu' does not support SELinux 2024-04-24T00:23:32.185 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T00:23:32.185 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-24T00:23:32.186 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T00:23:32.186 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T00:23:32.195 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T00:23:32.212 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T00:23:32.213 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T00:30:38.605 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2024-04-24T00:30:38.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-24T00:30:38.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:30:38.690 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-04-24T00:30:38.770 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-24T00:30:38.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-24T00:30:38.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T00:30:38.851 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-04-24T00:30:38.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-04-24T00:30:38.926 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T00:30:38.938 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T00:30:38.938 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T00:30:38.939 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T00:30:38.941 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T00:30:38.941 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T00:30:38.969 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-24T00:30:38.969 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Command line: ntpd -gq 2024-04-24T00:30:38.969 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: ---------------------------------------------------- 2024-04-24T00:30:38.970 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: ntp-4 is maintained by Network Time Foundation, 2024-04-24T00:30:38.970 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-24T00:30:38.970 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: corporation. Support and training for ntp-4 are 2024-04-24T00:30:38.970 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: available at https://www.nwtime.org/support 2024-04-24T00:30:38.970 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: ---------------------------------------------------- 2024-04-24T00:30:38.971 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: proto: precision = 0.068 usec (-24) 2024-04-24T00:30:38.971 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: basedate set to 2022-02-04 2024-04-24T00:30:38.971 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: gps base set to 2022-02-06 (week 2196) 2024-04-24T00:30:38.971 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T00:30:38.971 INFO:teuthology.orchestra.run.smithi064.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T00:30:38.971 INFO:teuthology.orchestra.run.smithi064.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T00:30:38.972 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T00:30:38.973 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T00:30:38.973 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T00:30:38.973 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T00:30:38.973 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Listen normally on 3 enp3s0f1 172.21.15.64:123 2024-04-24T00:30:38.973 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Listen normally on 4 lo [::1]:123 2024-04-24T00:30:38.973 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8df%5]:123 2024-04-24T00:30:38.973 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:38 ntpd[17198]: Listening on routing socket on fd #22 for interface updates 2024-04-24T00:30:38.996 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-24T00:30:38.996 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Command line: ntpd -gq 2024-04-24T00:30:38.996 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: ---------------------------------------------------- 2024-04-24T00:30:38.997 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: ntp-4 is maintained by Network Time Foundation, 2024-04-24T00:30:38.997 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-24T00:30:38.997 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: corporation. Support and training for ntp-4 are 2024-04-24T00:30:38.997 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: available at https://www.nwtime.org/support 2024-04-24T00:30:38.997 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: ---------------------------------------------------- 2024-04-24T00:30:38.997 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: proto: precision = 0.043 usec (-24) 2024-04-24T00:30:38.997 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: basedate set to 2022-02-04 2024-04-24T00:30:38.997 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: gps base set to 2022-02-06 (week 2196) 2024-04-24T00:30:38.998 INFO:teuthology.orchestra.run.smithi167.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T00:30:38.998 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T00:30:38.998 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T00:30:38.998 INFO:teuthology.orchestra.run.smithi167.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T00:30:38.999 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T00:30:38.999 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T00:30:38.999 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T00:30:39.000 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Listen normally on 3 enp3s0f1 172.21.15.167:123 2024-04-24T00:30:39.000 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Listen normally on 4 lo [::1]:123 2024-04-24T00:30:39.000 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7309%5]:123 2024-04-24T00:30:39.000 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:38 ntpd[17205]: Listening on routing socket on fd #22 for interface updates 2024-04-24T00:30:39.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:39 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:39.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:39 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:39.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:39 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:39.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:39 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:39.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:39 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:39.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:39 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:39.996 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:39 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:39.996 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:39 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:39.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:39 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:39.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:39 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:39.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:39 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:39.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:39 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:40.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:40 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:40.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:40 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:40.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:40 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:40.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:40 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:40.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:40 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:40.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:40 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:41.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:41.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:41.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:41.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:41.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:41.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:41.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:41.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:41 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:41.996 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:41 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:41.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:41 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:41.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:41 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:41.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:41 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:41.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:41 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:41.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:41 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:42 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:42 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:42 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:42 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:42 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:42 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:42.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:42 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:42.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:42 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:42.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:42 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:42.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:42 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:42.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:42 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:42.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:42 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:43 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:43.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:43 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:44.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:44 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:44.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:44 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:44.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:44 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:44.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:44 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:44.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:44 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:44.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:44 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:44.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:44 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:44.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:44 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:44.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:44 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:44.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:44 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:44.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:44 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:44.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:44 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:45.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:45.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:45.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:45.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:45.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:45.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:45.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:45.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:45.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:45.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:45.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:45.972 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:45 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:45.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:45.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:45.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:45.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:45.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:45.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:45.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:45.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:45.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:45.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:45.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:45.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:45 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:46.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:46 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:46.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:46 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:46.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:46 ntpd[17198]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:46.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:46 ntpd[17198]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:46.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:46 ntpd[17198]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:46.970 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:46 ntpd[17198]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:46.971 INFO:teuthology.orchestra.run.smithi064.stdout:24 Apr 00:30:46 ntpd[17198]: ntpd: time slew -0.001177 s 2024-04-24T00:30:46.971 INFO:teuthology.orchestra.run.smithi064.stdout:ntpd: time slew -0.001177s 2024-04-24T00:30:46.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:46 ntpd[17198]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T00:30:46.971 INFO:teuthology.orchestra.run.smithi064.stderr:24 Apr 00:30:46 ntpd[17198]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T00:30:46.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:46 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:46.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:46 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:46.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:46 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:46.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:46 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:46.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:46 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:46.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:46 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T00:30:47.121 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2024-04-24T00:30:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:30:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:30:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:30:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:30:47.997 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T00:30:47.998 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T00:30:47.999 INFO:teuthology.orchestra.run.smithi167.stdout:24 Apr 00:30:47 ntpd[17205]: ntpd: time slew -0.002811 s 2024-04-24T00:30:47.999 INFO:teuthology.orchestra.run.smithi167.stdout:ntpd: time slew -0.002811s 2024-04-24T00:30:47.999 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T00:30:47.999 INFO:teuthology.orchestra.run.smithi167.stderr:24 Apr 00:30:47 ntpd[17205]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T00:30:48.054 INFO:teuthology.orchestra.run.smithi167.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T00:30:48.054 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================== 2024-04-24T00:30:48.055 INFO:teuthology.orchestra.run.smithi167.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:30:48.055 INFO:teuthology.orchestra.run.smithi167.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:30:48.055 INFO:teuthology.orchestra.run.smithi167.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:30:48.055 INFO:teuthology.orchestra.run.smithi167.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:30:48.055 INFO:teuthology.run_tasks:Running task install... 2024-04-24T00:30:48.064 DEBUG:teuthology.task.install:project ceph 2024-04-24T00:30:48.064 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}} 2024-04-24T00:30:48.064 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'} 2024-04-24T00:30:48.064 INFO:teuthology.task.install:Using flavor: default 2024-04-24T00:30:48.070 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T00:30:48.070 INFO:teuthology.task.install:extra packages: [] 2024-04-24T00:30:48.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-key list | grep Ceph 2024-04-24T00:30:48.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-key list | grep Ceph 2024-04-24T00:30:48.137 INFO:teuthology.orchestra.run.smithi064.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-24T00:30:48.151 INFO:teuthology.orchestra.run.smithi167.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-24T00:30:48.168 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T00:30:48.169 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T00:30:48.170 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-24T00:30:48.170 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:30:48.179 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T00:30:48.179 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T00:30:48.179 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-24T00:30:48.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:30:48.322 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default/ 2024-04-24T00:30:48.322 INFO:teuthology.task.install.deb:Package version is 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:30:48.328 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default/ 2024-04-24T00:30:48.328 INFO:teuthology.task.install.deb:Package version is 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:30:48.705 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:30:48.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T00:30:48.707 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:30:48.707 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T00:30:48.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2024-04-24T00:30:48.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2024-04-24T00:30:48.899 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:30:48.900 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T00:30:48.910 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T00:30:48.930 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T00:30:49.017 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T00:30:49.017 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T00:30:49.088 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T00:30:49.138 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy InRelease 2024-04-24T00:30:49.152 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy InRelease 2024-04-24T00:30:49.170 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T00:30:49.209 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-24T00:30:49.223 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-24T00:30:49.282 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-24T00:30:49.296 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-24T00:30:49.353 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-24T00:30:49.368 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-24T00:30:49.431 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-24T00:30:49.448 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-24T00:30:49.521 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 28.9 kB in 1s (47.7 kB/s) 2024-04-24T00:30:49.542 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 28.9 kB in 1s (46.9 kB/s) 2024-04-24T00:30:50.502 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-04-24T00:30:50.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2424-gb98aab6c-1jammy cephadm=19.0.0-2424-gb98aab6c-1jammy ceph-mds=19.0.0-2424-gb98aab6c-1jammy ceph-mgr=19.0.0-2424-gb98aab6c-1jammy ceph-common=19.0.0-2424-gb98aab6c-1jammy ceph-fuse=19.0.0-2424-gb98aab6c-1jammy ceph-test=19.0.0-2424-gb98aab6c-1jammy ceph-volume=19.0.0-2424-gb98aab6c-1jammy radosgw=19.0.0-2424-gb98aab6c-1jammy python3-rados=19.0.0-2424-gb98aab6c-1jammy python3-rgw=19.0.0-2424-gb98aab6c-1jammy python3-cephfs=19.0.0-2424-gb98aab6c-1jammy python3-rbd=19.0.0-2424-gb98aab6c-1jammy libcephfs2=19.0.0-2424-gb98aab6c-1jammy libcephfs-dev=19.0.0-2424-gb98aab6c-1jammy librados2=19.0.0-2424-gb98aab6c-1jammy librbd1=19.0.0-2424-gb98aab6c-1jammy rbd-fuse=19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:30:50.546 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-04-24T00:30:50.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2424-gb98aab6c-1jammy cephadm=19.0.0-2424-gb98aab6c-1jammy ceph-mds=19.0.0-2424-gb98aab6c-1jammy ceph-mgr=19.0.0-2424-gb98aab6c-1jammy ceph-common=19.0.0-2424-gb98aab6c-1jammy ceph-fuse=19.0.0-2424-gb98aab6c-1jammy ceph-test=19.0.0-2424-gb98aab6c-1jammy ceph-volume=19.0.0-2424-gb98aab6c-1jammy radosgw=19.0.0-2424-gb98aab6c-1jammy python3-rados=19.0.0-2424-gb98aab6c-1jammy python3-rgw=19.0.0-2424-gb98aab6c-1jammy python3-cephfs=19.0.0-2424-gb98aab6c-1jammy python3-rbd=19.0.0-2424-gb98aab6c-1jammy libcephfs2=19.0.0-2424-gb98aab6c-1jammy libcephfs-dev=19.0.0-2424-gb98aab6c-1jammy librados2=19.0.0-2424-gb98aab6c-1jammy librbd1=19.0.0-2424-gb98aab6c-1jammy rbd-fuse=19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:30:50.572 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2024-04-24T00:30:50.626 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-04-24T00:30:50.748 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2024-04-24T00:30:50.748 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2024-04-24T00:30:50.830 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2024-04-24T00:30:50.831 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2024-04-24T00:30:50.922 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:30:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:30:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:30:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:30:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T00:30:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T00:30:50.923 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:30:50.925 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2024-04-24T00:30:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T00:30:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-24T00:30:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T00:30:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T00:30:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet zip 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-24T00:30:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-24T00:30:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-24T00:30:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-24T00:30:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-24T00:30:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T00:30:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-24T00:30:50.928 INFO:teuthology.orchestra.run.smithi167.stdout:Recommended packages: 2024-04-24T00:30:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: btrfs-tools 2024-04-24T00:30:51.027 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2024-04-24T00:30:51.027 INFO:teuthology.orchestra.run.smithi167.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T00:30:51.027 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T00:30:51.027 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-24T00:30:51.027 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-24T00:30:51.027 INFO:teuthology.orchestra.run.smithi167.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T00:30:51.027 INFO:teuthology.orchestra.run.smithi167.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T00:30:51.029 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-24T00:30:51.030 INFO:teuthology.orchestra.run.smithi167.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-24T00:30:51.036 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T00:30:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T00:30:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T00:30:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T00:30:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T00:30:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T00:30:51.037 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T00:30:51.037 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2024-04-24T00:30:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T00:30:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T00:30:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-24T00:30:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet zip 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2024-04-24T00:30:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: btrfs-tools 2024-04-24T00:30:51.095 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-24T00:30:51.095 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 179 MB of archives. 2024-04-24T00:30:51.095 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-24T00:30:51.095 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-24T00:30:51.146 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2024-04-24T00:30:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T00:30:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T00:30:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-24T00:30:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-24T00:30:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T00:30:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-24T00:30:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-24T00:30:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-24T00:30:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-24T00:30:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-24T00:30:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-24T00:30:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-24T00:30:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-24T00:30:51.201 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-24T00:30:51.213 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-24T00:30:51.215 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-24T00:30:51.215 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 179 MB of archives. 2024-04-24T00:30:51.215 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-24T00:30:51.215 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-24T00:30:51.233 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-24T00:30:51.298 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-24T00:30:51.303 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-24T00:30:51.316 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-24T00:30:51.320 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-24T00:30:51.320 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-24T00:30:51.321 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-24T00:30:51.322 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-24T00:30:51.325 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-24T00:30:51.330 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-24T00:30:51.332 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-24T00:30:51.334 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-24T00:30:51.336 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-24T00:30:51.343 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-24T00:30:51.346 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-24T00:30:51.347 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-24T00:30:51.348 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-24T00:30:51.364 INFO:teuthology.orchestra.run.smithi167.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-24T00:30:51.365 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-24T00:30:51.365 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-24T00:30:51.367 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-24T00:30:51.367 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-24T00:30:51.367 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-24T00:30:51.367 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-24T00:30:51.385 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-24T00:30:51.385 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-24T00:30:51.386 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-24T00:30:51.405 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-24T00:30:51.406 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-24T00:30:51.406 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-24T00:30:51.407 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-24T00:30:51.408 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T00:30:51.408 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-24T00:30:51.408 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-24T00:30:51.414 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-24T00:30:51.418 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-24T00:30:51.428 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-24T00:30:51.428 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-24T00:30:51.429 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-24T00:30:51.431 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-24T00:30:51.436 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-24T00:30:51.436 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-24T00:30:51.437 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-24T00:30:51.446 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-24T00:30:51.449 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-24T00:30:51.449 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-24T00:30:51.449 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-24T00:30:51.450 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-24T00:30:51.450 INFO:teuthology.orchestra.run.smithi167.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-24T00:30:51.451 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-24T00:30:51.452 INFO:teuthology.orchestra.run.smithi167.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-24T00:30:51.452 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-24T00:30:51.454 INFO:teuthology.orchestra.run.smithi167.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-24T00:30:51.454 INFO:teuthology.orchestra.run.smithi167.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-24T00:30:51.459 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-24T00:30:51.461 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-24T00:30:51.463 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-24T00:30:51.469 INFO:teuthology.orchestra.run.smithi167.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-24T00:30:51.470 INFO:teuthology.orchestra.run.smithi167.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-24T00:30:51.473 INFO:teuthology.orchestra.run.smithi167.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-24T00:30:51.479 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-24T00:30:51.480 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-24T00:30:51.481 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-24T00:30:51.482 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-24T00:30:51.482 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-24T00:30:51.483 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-24T00:30:51.483 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-24T00:30:51.490 INFO:teuthology.orchestra.run.smithi167.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-24T00:30:51.491 INFO:teuthology.orchestra.run.smithi167.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-24T00:30:51.494 INFO:teuthology.orchestra.run.smithi167.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-24T00:30:51.513 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-24T00:30:51.513 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-24T00:30:51.514 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-24T00:30:51.535 INFO:teuthology.orchestra.run.smithi167.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2424-gb98aab6c-1jammy [3,544 kB] 2024-04-24T00:30:51.542 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-24T00:30:51.543 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-24T00:30:51.543 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-24T00:30:51.543 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-24T00:30:51.545 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T00:30:51.545 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-24T00:30:51.545 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-24T00:30:51.547 INFO:teuthology.orchestra.run.smithi167.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-24T00:30:51.547 INFO:teuthology.orchestra.run.smithi167.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-24T00:30:51.549 INFO:teuthology.orchestra.run.smithi167.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T00:30:51.549 INFO:teuthology.orchestra.run.smithi167.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-24T00:30:51.550 INFO:teuthology.orchestra.run.smithi167.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-24T00:30:51.567 INFO:teuthology.orchestra.run.smithi167.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-24T00:30:51.569 INFO:teuthology.orchestra.run.smithi167.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-24T00:30:51.569 INFO:teuthology.orchestra.run.smithi167.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-24T00:30:51.570 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-24T00:30:51.570 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-24T00:30:51.571 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-24T00:30:51.590 INFO:teuthology.orchestra.run.smithi167.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-24T00:30:51.590 INFO:teuthology.orchestra.run.smithi167.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-24T00:30:51.591 INFO:teuthology.orchestra.run.smithi167.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-24T00:30:51.591 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-24T00:30:51.592 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-24T00:30:51.592 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-24T00:30:51.593 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-24T00:30:51.594 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-24T00:30:51.596 INFO:teuthology.orchestra.run.smithi064.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-24T00:30:51.597 INFO:teuthology.orchestra.run.smithi064.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-24T00:30:51.601 INFO:teuthology.orchestra.run.smithi167.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-24T00:30:51.601 INFO:teuthology.orchestra.run.smithi167.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-24T00:30:51.602 INFO:teuthology.orchestra.run.smithi167.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-24T00:30:51.602 INFO:teuthology.orchestra.run.smithi167.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-24T00:30:51.602 INFO:teuthology.orchestra.run.smithi167.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-24T00:30:51.606 INFO:teuthology.orchestra.run.smithi167.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-24T00:30:51.612 INFO:teuthology.orchestra.run.smithi064.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-24T00:30:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-24T00:30:51.615 INFO:teuthology.orchestra.run.smithi064.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-24T00:30:51.621 INFO:teuthology.orchestra.run.smithi167.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-24T00:30:51.623 INFO:teuthology.orchestra.run.smithi167.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-24T00:30:51.623 INFO:teuthology.orchestra.run.smithi167.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-24T00:30:51.626 INFO:teuthology.orchestra.run.smithi167.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-24T00:30:51.633 INFO:teuthology.orchestra.run.smithi064.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-24T00:30:51.633 INFO:teuthology.orchestra.run.smithi064.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-24T00:30:51.636 INFO:teuthology.orchestra.run.smithi064.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-24T00:30:51.656 INFO:teuthology.orchestra.run.smithi167.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-24T00:30:51.660 INFO:teuthology.orchestra.run.smithi167.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-24T00:30:51.661 INFO:teuthology.orchestra.run.smithi167.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-24T00:30:51.666 INFO:teuthology.orchestra.run.smithi167.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-24T00:30:51.667 INFO:teuthology.orchestra.run.smithi167.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-24T00:30:51.670 INFO:teuthology.orchestra.run.smithi064.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2424-gb98aab6c-1jammy [3,544 kB] 2024-04-24T00:30:51.671 INFO:teuthology.orchestra.run.smithi167.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-24T00:30:51.689 INFO:teuthology.orchestra.run.smithi167.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T00:30:51.690 INFO:teuthology.orchestra.run.smithi167.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-24T00:30:51.690 INFO:teuthology.orchestra.run.smithi167.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-24T00:30:51.692 INFO:teuthology.orchestra.run.smithi167.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-24T00:30:51.693 INFO:teuthology.orchestra.run.smithi064.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-24T00:30:51.694 INFO:teuthology.orchestra.run.smithi064.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-24T00:30:51.695 INFO:teuthology.orchestra.run.smithi064.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T00:30:51.695 INFO:teuthology.orchestra.run.smithi064.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-24T00:30:51.696 INFO:teuthology.orchestra.run.smithi064.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-24T00:30:51.715 INFO:teuthology.orchestra.run.smithi064.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-24T00:30:51.717 INFO:teuthology.orchestra.run.smithi064.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-24T00:30:51.717 INFO:teuthology.orchestra.run.smithi064.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-24T00:30:51.718 INFO:teuthology.orchestra.run.smithi167.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-24T00:30:51.727 INFO:teuthology.orchestra.run.smithi167.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-24T00:30:51.727 INFO:teuthology.orchestra.run.smithi167.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-24T00:30:51.727 INFO:teuthology.orchestra.run.smithi167.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-24T00:30:51.730 INFO:teuthology.orchestra.run.smithi167.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-24T00:30:51.731 INFO:teuthology.orchestra.run.smithi167.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-24T00:30:51.764 INFO:teuthology.orchestra.run.smithi064.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-24T00:30:51.765 INFO:teuthology.orchestra.run.smithi064.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-24T00:30:51.765 INFO:teuthology.orchestra.run.smithi064.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-24T00:30:51.766 INFO:teuthology.orchestra.run.smithi064.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-24T00:30:51.767 INFO:teuthology.orchestra.run.smithi064.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-24T00:30:51.768 INFO:teuthology.orchestra.run.smithi064.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-24T00:30:51.768 INFO:teuthology.orchestra.run.smithi064.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-24T00:30:51.769 INFO:teuthology.orchestra.run.smithi064.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-24T00:30:51.774 INFO:teuthology.orchestra.run.smithi064.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-24T00:30:51.786 INFO:teuthology.orchestra.run.smithi064.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-24T00:30:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-24T00:30:51.798 INFO:teuthology.orchestra.run.smithi064.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-24T00:30:51.802 INFO:teuthology.orchestra.run.smithi064.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-24T00:30:51.806 INFO:teuthology.orchestra.run.smithi064.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-24T00:30:51.807 INFO:teuthology.orchestra.run.smithi064.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-24T00:30:51.807 INFO:teuthology.orchestra.run.smithi064.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-24T00:30:51.814 INFO:teuthology.orchestra.run.smithi064.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-24T00:30:51.814 INFO:teuthology.orchestra.run.smithi064.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-24T00:30:51.818 INFO:teuthology.orchestra.run.smithi064.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-24T00:30:51.833 INFO:teuthology.orchestra.run.smithi064.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T00:30:51.834 INFO:teuthology.orchestra.run.smithi064.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-24T00:30:51.834 INFO:teuthology.orchestra.run.smithi064.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-24T00:30:51.836 INFO:teuthology.orchestra.run.smithi064.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-24T00:30:51.837 INFO:teuthology.orchestra.run.smithi064.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-24T00:30:51.846 INFO:teuthology.orchestra.run.smithi064.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-24T00:30:51.853 INFO:teuthology.orchestra.run.smithi064.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-24T00:30:51.854 INFO:teuthology.orchestra.run.smithi064.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-24T00:30:51.856 INFO:teuthology.orchestra.run.smithi064.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-24T00:30:51.857 INFO:teuthology.orchestra.run.smithi064.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-24T00:30:52.214 INFO:teuthology.orchestra.run.smithi064.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2424-gb98aab6c-1jammy [3,218 kB] 2024-04-24T00:30:52.383 INFO:teuthology.orchestra.run.smithi064.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2424-gb98aab6c-1jammy [711 kB] 2024-04-24T00:30:52.429 INFO:teuthology.orchestra.run.smithi064.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2424-gb98aab6c-1jammy [356 kB] 2024-04-24T00:30:52.442 INFO:teuthology.orchestra.run.smithi064.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2424-gb98aab6c-1jammy [32.8 kB] 2024-04-24T00:30:52.443 INFO:teuthology.orchestra.run.smithi064.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2424-gb98aab6c-1jammy [180 kB] 2024-04-24T00:30:52.449 INFO:teuthology.orchestra.run.smithi064.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2424-gb98aab6c-1jammy [65.7 kB] 2024-04-24T00:30:52.451 INFO:teuthology.orchestra.run.smithi064.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2424-gb98aab6c-1jammy [330 kB] 2024-04-24T00:30:52.464 INFO:teuthology.orchestra.run.smithi064.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2424-gb98aab6c-1jammy [6,775 kB] 2024-04-24T00:30:52.820 INFO:teuthology.orchestra.run.smithi167.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2424-gb98aab6c-1jammy [3,218 kB] 2024-04-24T00:30:52.842 INFO:teuthology.orchestra.run.smithi064.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2424-gb98aab6c-1jammy [112 kB] 2024-04-24T00:30:52.845 INFO:teuthology.orchestra.run.smithi064.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2424-gb98aab6c-1jammy [466 kB] 2024-04-24T00:30:52.859 INFO:teuthology.orchestra.run.smithi064.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2424-gb98aab6c-1jammy [25.6 MB] 2024-04-24T00:30:53.543 INFO:teuthology.orchestra.run.smithi167.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2424-gb98aab6c-1jammy [711 kB] 2024-04-24T00:30:53.687 INFO:teuthology.orchestra.run.smithi167.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2424-gb98aab6c-1jammy [356 kB] 2024-04-24T00:30:53.760 INFO:teuthology.orchestra.run.smithi167.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2424-gb98aab6c-1jammy [32.8 kB] 2024-04-24T00:30:53.761 INFO:teuthology.orchestra.run.smithi167.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2424-gb98aab6c-1jammy [180 kB] 2024-04-24T00:30:53.828 INFO:teuthology.orchestra.run.smithi167.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2424-gb98aab6c-1jammy [65.7 kB] 2024-04-24T00:30:53.830 INFO:teuthology.orchestra.run.smithi167.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2424-gb98aab6c-1jammy [330 kB] 2024-04-24T00:30:53.903 INFO:teuthology.orchestra.run.smithi167.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2424-gb98aab6c-1jammy [6,775 kB] 2024-04-24T00:30:54.242 INFO:teuthology.orchestra.run.smithi064.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2424-gb98aab6c-1jammy [5,084 kB] 2024-04-24T00:30:54.401 INFO:teuthology.orchestra.run.smithi064.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2424-gb98aab6c-1jammy [244 kB] 2024-04-24T00:30:54.407 INFO:teuthology.orchestra.run.smithi064.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2424-gb98aab6c-1jammy [124 kB] 2024-04-24T00:30:54.412 INFO:teuthology.orchestra.run.smithi064.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2424-gb98aab6c-1jammy [1,526 kB] 2024-04-24T00:30:54.488 INFO:teuthology.orchestra.run.smithi064.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2424-gb98aab6c-1jammy [6,246 kB] 2024-04-24T00:30:54.803 INFO:teuthology.orchestra.run.smithi064.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2424-gb98aab6c-1jammy [22.7 MB] 2024-04-24T00:30:55.422 INFO:teuthology.orchestra.run.smithi167.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2424-gb98aab6c-1jammy [112 kB] 2024-04-24T00:30:55.424 INFO:teuthology.orchestra.run.smithi167.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2424-gb98aab6c-1jammy [466 kB] 2024-04-24T00:30:55.503 INFO:teuthology.orchestra.run.smithi167.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2424-gb98aab6c-1jammy [25.6 MB] 2024-04-24T00:30:56.044 INFO:teuthology.orchestra.run.smithi064.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2424-gb98aab6c-1jammy [14.0 kB] 2024-04-24T00:30:56.044 INFO:teuthology.orchestra.run.smithi064.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2424-gb98aab6c-1jammy [869 kB] 2024-04-24T00:30:56.062 INFO:teuthology.orchestra.run.smithi064.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2424-gb98aab6c-1jammy [2,437 kB] 2024-04-24T00:30:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2424-gb98aab6c-1jammy [790 kB] 2024-04-24T00:30:56.156 INFO:teuthology.orchestra.run.smithi064.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2424-gb98aab6c-1jammy [147 kB] 2024-04-24T00:30:56.160 INFO:teuthology.orchestra.run.smithi064.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2424-gb98aab6c-1jammy [2,295 kB] 2024-04-24T00:30:56.297 INFO:teuthology.orchestra.run.smithi064.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2424-gb98aab6c-1jammy [8,625 kB] 2024-04-24T00:30:56.709 INFO:teuthology.orchestra.run.smithi064.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2424-gb98aab6c-1jammy [14.2 kB] 2024-04-24T00:30:56.733 INFO:teuthology.orchestra.run.smithi064.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2424-gb98aab6c-1jammy [50.7 MB] 2024-04-24T00:30:59.432 INFO:teuthology.orchestra.run.smithi064.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2424-gb98aab6c-1jammy [121 kB] 2024-04-24T00:30:59.434 INFO:teuthology.orchestra.run.smithi064.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2424-gb98aab6c-1jammy [38.9 kB] 2024-04-24T00:30:59.435 INFO:teuthology.orchestra.run.smithi064.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2424-gb98aab6c-1jammy [13.4 MB] 2024-04-24T00:30:59.949 INFO:teuthology.orchestra.run.smithi064.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2424-gb98aab6c-1jammy [91.4 kB] 2024-04-24T00:31:00.337 INFO:teuthology.orchestra.run.smithi167.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2424-gb98aab6c-1jammy [5,084 kB] 2024-04-24T00:31:00.546 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 179 MB in 9s (20.4 MB/s) 2024-04-24T00:31:00.680 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-24T00:31:00.701 INFO:teuthology.orchestra.run.smithi167.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2424-gb98aab6c-1jammy [244 kB] 2024-04-24T00:31:00.731 INFO:teuthology.orchestra.run.smithi167.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2424-gb98aab6c-1jammy [124 kB] 2024-04-24T00:31:00.734 INFO:teuthology.orchestra.run.smithi167.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2424-gb98aab6c-1jammy [1,526 kB] 2024-04-24T00:31:00.781 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-24T00:31:00.787 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-24T00:31:00.847 INFO:teuthology.orchestra.run.smithi167.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2424-gb98aab6c-1jammy [6,246 kB] 2024-04-24T00:31:00.853 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T00:31:01.123 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T00:31:01.150 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-24T00:31:01.181 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T00:31:01.376 INFO:teuthology.orchestra.run.smithi167.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2424-gb98aab6c-1jammy [22.7 MB] 2024-04-24T00:31:01.526 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T00:31:01.553 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-24T00:31:01.593 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T00:31:02.038 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T00:31:02.065 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:31:02.113 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:31:02.449 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T00:31:02.476 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:31:02.507 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:31:02.861 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T00:31:02.887 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:31:02.927 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:31:03.264 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-24T00:31:03.279 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-24T00:31:03.301 INFO:teuthology.orchestra.run.smithi167.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2424-gb98aab6c-1jammy [14.0 kB] 2024-04-24T00:31:03.302 INFO:teuthology.orchestra.run.smithi167.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2424-gb98aab6c-1jammy [869 kB] 2024-04-24T00:31:03.313 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T00:31:03.318 INFO:teuthology.orchestra.run.smithi167.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2424-gb98aab6c-1jammy [2,437 kB] 2024-04-24T00:31:03.513 INFO:teuthology.orchestra.run.smithi167.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2424-gb98aab6c-1jammy [790 kB] 2024-04-24T00:31:03.556 INFO:teuthology.orchestra.run.smithi167.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2424-gb98aab6c-1jammy [147 kB] 2024-04-24T00:31:03.584 INFO:teuthology.orchestra.run.smithi167.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2424-gb98aab6c-1jammy [2,295 kB] 2024-04-24T00:31:03.599 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librados2. 2024-04-24T00:31:03.616 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../007-librados2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:03.658 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librados2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:03.767 INFO:teuthology.orchestra.run.smithi167.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2424-gb98aab6c-1jammy [8,625 kB] 2024-04-24T00:31:04.135 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librbd1. 2024-04-24T00:31:04.163 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../008-librbd1_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:04.202 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librbd1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:04.389 INFO:teuthology.orchestra.run.smithi167.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2424-gb98aab6c-1jammy [14.2 kB] 2024-04-24T00:31:04.389 INFO:teuthology.orchestra.run.smithi167.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2424-gb98aab6c-1jammy [50.7 MB] 2024-04-24T00:31:04.647 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T00:31:04.674 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:04.705 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:04.982 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rados. 2024-04-24T00:31:05.009 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:05.049 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rados (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:05.377 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T00:31:05.404 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:05.443 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-argparse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:05.679 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T00:31:05.695 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:05.746 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cephfs (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:06.032 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T00:31:06.048 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:06.082 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:06.368 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T00:31:06.384 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-24T00:31:06.418 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T00:31:06.746 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T00:31:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-24T00:31:06.821 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-24T00:31:07.090 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T00:31:07.106 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:07.266 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rbd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:07.693 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T00:31:07.720 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-24T00:31:07.759 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T00:31:07.828 INFO:teuthology.orchestra.run.smithi167.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2424-gb98aab6c-1jammy [121 kB] 2024-04-24T00:31:07.830 INFO:teuthology.orchestra.run.smithi167.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2424-gb98aab6c-1jammy [38.9 kB] 2024-04-24T00:31:07.831 INFO:teuthology.orchestra.run.smithi167.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2424-gb98aab6c-1jammy [13.4 MB] 2024-04-24T00:31:08.012 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-24T00:31:08.039 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-24T00:31:08.079 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T00:31:08.347 INFO:teuthology.orchestra.run.smithi167.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2424-gb98aab6c-1jammy [91.4 kB] 2024-04-24T00:31:08.390 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-24T00:31:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-24T00:31:08.448 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T00:31:08.776 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package lua5.1. 2024-04-24T00:31:08.803 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-24T00:31:08.843 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-24T00:31:08.899 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 179 MB in 17s (10.4 MB/s) 2024-04-24T00:31:09.070 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-24T00:31:09.137 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package lua-any. 2024-04-24T00:31:09.153 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-24T00:31:09.171 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-24T00:31:09.177 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-24T00:31:09.196 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-24T00:31:09.268 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T00:31:09.440 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package zip. 2024-04-24T00:31:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-24T00:31:09.498 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking zip (3.0-12build2) ... 2024-04-24T00:31:09.614 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T00:31:09.640 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-24T00:31:09.680 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T00:31:09.827 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package unzip. 2024-04-24T00:31:09.854 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-24T00:31:09.893 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-24T00:31:10.008 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T00:31:10.035 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-24T00:31:10.075 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T00:31:10.291 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package luarocks. 2024-04-24T00:31:10.318 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-24T00:31:10.349 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-24T00:31:10.453 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T00:31:10.480 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:31:10.528 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librgw2. 2024-04-24T00:31:10.692 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../025-librgw2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:10.722 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librgw2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:10.932 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:31:10.998 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:31:11.267 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T00:31:11.283 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:11.368 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T00:31:11.394 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T00:31:11.435 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:31:11.661 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-24T00:31:11.688 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-24T00:31:11.729 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T00:31:11.747 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-24T00:31:11.779 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T00:31:12.014 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T00:31:12.041 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:12.072 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libradosstriper1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:12.091 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librados2. 2024-04-24T00:31:12.117 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../007-librados2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:12.157 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librados2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:12.534 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-common. 2024-04-24T00:31:12.561 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:12.600 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:12.827 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librbd1. 2024-04-24T00:31:12.853 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../008-librbd1_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:12.893 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librbd1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:13.338 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T00:31:13.353 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:13.388 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libcephfs2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:13.649 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rados. 2024-04-24T00:31:13.675 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:13.715 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rados (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:13.998 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-base. 2024-04-24T00:31:14.025 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:14.052 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T00:31:14.073 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-base (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:14.078 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:14.119 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-ceph-argparse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:14.371 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T00:31:14.386 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:14.429 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cephfs (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:14.635 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-24T00:31:14.662 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-24T00:31:14.702 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-24T00:31:14.707 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T00:31:14.733 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:14.765 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:15.004 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cheroot. 2024-04-24T00:31:15.031 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-24T00:31:15.035 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T00:31:15.062 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-24T00:31:15.071 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T00:31:15.093 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T00:31:15.413 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T00:31:15.415 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-24T00:31:15.440 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-24T00:31:15.443 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-24T00:31:15.480 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-24T00:31:15.482 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-24T00:31:15.718 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-24T00:31:15.745 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-24T00:31:15.749 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T00:31:15.776 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:15.785 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-24T00:31:15.815 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rbd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:16.088 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-24T00:31:16.115 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-24T00:31:16.154 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-24T00:31:16.162 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T00:31:16.189 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-24T00:31:16.229 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T00:31:16.457 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-tempora. 2024-04-24T00:31:16.481 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-24T00:31:16.484 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-24T00:31:16.497 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-24T00:31:16.524 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-24T00:31:16.531 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T00:31:16.793 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-portend. 2024-04-24T00:31:16.821 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-24T00:31:16.842 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-24T00:31:16.851 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-24T00:31:16.858 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-24T00:31:16.917 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T00:31:17.196 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-24T00:31:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-24T00:31:17.263 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package lua5.1. 2024-04-24T00:31:17.270 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-24T00:31:17.289 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-24T00:31:17.329 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-24T00:31:17.623 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package lua-any. 2024-04-24T00:31:17.643 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-24T00:31:17.776 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T00:31:17.790 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-24T00:31:17.804 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-24T00:31:17.907 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-24T00:31:18.035 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package zip. 2024-04-24T00:31:18.050 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-24T00:31:18.093 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking zip (3.0-12build2) ... 2024-04-24T00:31:18.227 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T00:31:18.254 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-24T00:31:18.294 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-24T00:31:18.396 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package unzip. 2024-04-24T00:31:18.422 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-24T00:31:18.463 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-24T00:31:18.605 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T00:31:18.632 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-24T00:31:18.672 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-24T00:31:18.783 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package luarocks. 2024-04-24T00:31:18.809 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-24T00:31:18.841 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-24T00:31:18.983 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T00:31:19.010 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-24T00:31:19.049 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-24T00:31:19.224 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librgw2. 2024-04-24T00:31:19.251 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../025-librgw2_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:19.290 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librgw2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:19.327 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-mako. 2024-04-24T00:31:19.355 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-24T00:31:19.394 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T00:31:19.689 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T00:31:19.716 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T00:31:19.747 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T00:31:19.851 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T00:31:19.878 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:19.918 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:20.058 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T00:31:20.085 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-24T00:31:20.125 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-24T00:31:20.238 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-24T00:31:20.264 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-24T00:31:20.296 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T00:31:20.419 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webob. 2024-04-24T00:31:20.446 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-24T00:31:20.485 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-24T00:31:20.565 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T00:31:20.592 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:20.632 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libradosstriper1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:20.838 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T00:31:20.866 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-24T00:31:20.908 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T00:31:21.102 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-common. 2024-04-24T00:31:21.128 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:21.168 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:21.241 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T00:31:21.269 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-24T00:31:21.308 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T00:31:21.552 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-paste. 2024-04-24T00:31:21.579 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-24T00:31:21.610 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T00:31:21.938 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T00:31:21.955 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-24T00:31:21.997 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T00:31:22.258 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T00:31:22.274 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-24T00:31:22.316 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-24T00:31:22.566 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-base. 2024-04-24T00:31:22.593 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:22.619 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T00:31:22.642 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-base (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:22.646 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-24T00:31:23.203 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-24T00:31:23.230 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-24T00:31:23.230 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T00:31:23.258 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-24T00:31:23.269 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-24T00:31:23.581 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cheroot. 2024-04-24T00:31:23.607 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-24T00:31:23.639 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T00:31:23.648 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T00:31:23.959 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-24T00:31:23.985 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-24T00:31:24.025 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-24T00:31:24.244 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-certifi. 2024-04-24T00:31:24.271 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-24T00:31:24.310 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-24T00:31:24.638 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-idna. 2024-04-24T00:31:24.665 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-24T00:31:24.696 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-24T00:31:24.957 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-urllib3. 2024-04-24T00:31:24.985 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-24T00:31:25.023 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T00:31:25.178 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-24T00:31:25.205 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-24T00:31:25.245 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-24T00:31:25.319 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-requests. 2024-04-24T00:31:25.346 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-24T00:31:25.385 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T00:31:25.515 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-24T00:31:25.541 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-24T00:31:25.573 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-24T00:31:25.696 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T00:31:25.724 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-24T00:31:25.763 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T00:31:25.859 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-tempora. 2024-04-24T00:31:25.886 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-24T00:31:25.926 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-24T00:31:26.041 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T00:31:26.068 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:26.108 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:26.220 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-portend. 2024-04-24T00:31:26.246 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-24T00:31:26.286 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-24T00:31:26.513 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T00:31:26.539 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:26.579 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:26.590 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-24T00:31:26.616 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-24T00:31:26.656 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-24T00:31:26.849 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T00:31:26.876 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-24T00:31:26.906 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-24T00:31:26.959 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T00:31:26.985 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-24T00:31:27.026 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-24T00:31:27.209 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T00:31:27.237 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-24T00:31:27.276 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-24T00:31:27.321 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T00:31:27.336 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-24T00:31:27.388 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-24T00:31:27.621 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T00:31:27.648 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:27.687 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:27.699 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T00:31:27.725 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-24T00:31:27.765 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-24T00:31:28.040 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T00:31:28.056 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:28.090 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mon (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:28.310 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T00:31:28.337 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-24T00:31:28.376 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-24T00:31:28.688 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-mako. 2024-04-24T00:31:28.710 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T00:31:28.714 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-24T00:31:28.738 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:28.754 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T00:31:28.777 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-osd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:29.049 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T00:31:29.076 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T00:31:29.108 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T00:31:29.402 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T00:31:29.428 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-24T00:31:29.469 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-24T00:31:29.763 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webob. 2024-04-24T00:31:29.790 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-24T00:31:29.830 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-24T00:31:29.855 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph. 2024-04-24T00:31:29.883 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../066-ceph_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:29.921 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T00:31:30.234 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-24T00:31:30.235 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:30.274 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:30.277 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T00:31:30.569 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T00:31:30.595 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-24T00:31:30.602 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T00:31:30.630 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:30.635 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T00:31:30.669 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mds (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:30.913 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-paste. 2024-04-24T00:31:30.940 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-24T00:31:30.980 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T00:31:31.072 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package cephadm. 2024-04-24T00:31:31.099 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../069-cephadm_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:31.138 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:31.316 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T00:31:31.343 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-24T00:31:31.383 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T00:31:31.458 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-24T00:31:31.485 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-24T00:31:31.524 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-24T00:31:31.636 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T00:31:31.662 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-24T00:31:31.702 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-24T00:31:31.885 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-24T00:31:31.912 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T00:31:31.952 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T00:31:32.005 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T00:31:32.032 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-24T00:31:32.072 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-24T00:31:32.400 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T00:31:32.426 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-24T00:31:32.466 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T00:31:32.838 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-certifi. 2024-04-24T00:31:32.864 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-24T00:31:32.913 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-24T00:31:32.973 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-babel. 2024-04-24T00:31:33.001 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T00:31:33.031 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T00:31:33.224 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-idna. 2024-04-24T00:31:33.250 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-24T00:31:33.282 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-24T00:31:33.543 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-urllib3. 2024-04-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-24T00:31:33.595 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-24T00:31:33.601 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T00:31:33.635 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T00:31:33.938 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-requests. 2024-04-24T00:31:33.939 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:33.965 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-24T00:31:33.970 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:34.005 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T00:31:34.282 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T00:31:34.309 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T00:31:34.316 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T00:31:34.340 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T00:31:34.343 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-24T00:31:34.382 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T00:31:34.618 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-routes. 2024-04-24T00:31:34.646 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-24T00:31:34.677 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T00:31:34.684 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T00:31:34.693 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:34.735 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:34.988 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T00:31:35.015 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:35.079 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:35.107 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T00:31:35.134 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:35.174 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:35.460 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T00:31:35.487 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-24T00:31:35.526 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-24T00:31:35.829 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T00:31:35.856 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-24T00:31:35.896 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-24T00:31:36.266 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T00:31:36.292 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:36.332 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-24T00:31:36.393 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-24T00:31:36.434 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T00:31:36.710 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T00:31:36.737 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:36.768 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mon (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:36.946 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T00:31:36.974 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-24T00:31:37.012 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T00:31:37.347 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T00:31:37.374 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:37.399 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-24T00:31:37.413 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-osd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:37.427 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-24T00:31:37.466 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-24T00:31:37.810 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T00:31:37.838 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-24T00:31:37.885 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T00:31:38.611 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T00:31:38.626 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph. 2024-04-24T00:31:38.639 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:38.652 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../066-ceph_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:38.692 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:38.762 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:39.019 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T00:31:39.046 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:39.086 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:39.456 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T00:31:39.483 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:39.522 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mds (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:39.774 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T00:31:39.802 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-24T00:31:39.841 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-24T00:31:39.867 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package cephadm. 2024-04-24T00:31:39.894 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../069-cephadm_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:39.958 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:40.144 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T00:31:40.171 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-24T00:31:40.210 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-24T00:31:40.286 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-24T00:31:40.313 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-24T00:31:40.353 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-24T00:31:40.530 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T00:31:40.558 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-24T00:31:40.597 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T00:31:40.656 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-24T00:31:40.682 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T00:31:40.722 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T00:31:40.891 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T00:31:40.919 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-24T00:31:40.958 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-24T00:31:41.278 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T00:31:41.306 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-24T00:31:41.345 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-24T00:31:41.665 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T00:31:41.693 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-24T00:31:41.731 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T00:31:41.761 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-babel. 2024-04-24T00:31:41.788 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T00:31:41.827 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T00:31:42.017 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T00:31:42.045 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-24T00:31:42.076 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-24T00:31:42.146 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T00:31:42.174 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-24T00:31:42.213 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T00:31:42.354 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T00:31:42.382 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-24T00:31:42.420 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T00:31:42.566 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T00:31:42.594 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:42.633 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:42.970 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T00:31:43.036 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T00:31:43.126 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T00:31:43.154 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:43.192 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:43.339 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-routes. 2024-04-24T00:31:43.366 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-24T00:31:43.414 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T00:31:43.597 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T00:31:43.625 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-24T00:31:43.703 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T00:31:43.809 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T00:31:43.836 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:43.875 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:44.040 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T00:31:44.068 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T00:31:44.106 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T00:31:44.375 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package jq. 2024-04-24T00:31:44.392 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T00:31:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-24T00:31:44.686 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package socat. 2024-04-24T00:31:44.703 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-24T00:31:44.744 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-24T00:31:45.048 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T00:31:45.066 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-24T00:31:45.105 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-24T00:31:45.130 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-24T00:31:45.146 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-24T00:31:45.188 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T00:31:45.469 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-test. 2024-04-24T00:31:45.485 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:45.526 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-test (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:45.709 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T00:31:45.736 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-24T00:31:45.775 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T00:31:46.078 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-24T00:31:46.094 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-24T00:31:46.128 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-24T00:31:46.423 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T00:31:46.450 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-24T00:31:46.489 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T00:31:47.240 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T00:31:47.267 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:47.392 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:47.773 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T00:31:47.800 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:47.839 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-volume (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:48.151 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T00:31:48.168 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:48.209 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs-dev (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:48.329 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T00:31:48.356 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-24T00:31:48.386 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-24T00:31:48.512 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-24T00:31:48.540 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-24T00:31:48.579 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T00:31:48.681 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T00:31:48.708 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-24T00:31:48.748 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-24T00:31:49.141 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-24T00:31:49.169 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-24T00:31:49.207 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-24T00:31:49.260 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T00:31:49.287 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-24T00:31:49.326 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T00:31:49.580 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T00:31:49.608 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-24T00:31:49.637 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T00:31:49.665 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-24T00:31:49.672 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T00:31:49.703 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-24T00:31:50.025 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T00:31:50.052 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-24T00:31:50.090 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-24T00:31:50.093 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package pkg-config. 2024-04-24T00:31:50.122 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-24T00:31:50.159 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T00:31:50.411 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T00:31:50.421 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-24T00:31:50.438 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-24T00:31:50.449 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-24T00:31:50.477 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T00:31:50.487 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-24T00:31:50.797 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T00:31:50.824 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-24T00:31:50.863 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-24T00:31:50.890 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-24T00:31:50.907 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-24T00:31:50.940 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-24T00:31:51.150 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T00:31:51.177 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-24T00:31:51.208 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T00:31:51.243 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T00:31:51.271 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T00:31:51.309 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T00:31:51.638 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T00:31:51.666 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-24T00:31:51.704 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-24T00:31:51.905 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T00:31:51.922 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:51.963 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T00:31:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-24T00:31:52.082 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-24T00:31:52.258 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T00:31:52.286 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-24T00:31:52.325 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T00:31:52.451 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-py. 2024-04-24T00:31:52.479 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-24T00:31:52.526 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-24T00:31:52.619 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T00:31:52.647 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T00:31:52.686 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T00:31:52.863 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pygments. 2024-04-24T00:31:52.891 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-24T00:31:52.929 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T00:31:52.955 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package jq. 2024-04-24T00:31:52.971 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T00:31:53.013 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-24T00:31:53.249 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package socat. 2024-04-24T00:31:53.265 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-24T00:31:53.307 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-24T00:31:53.408 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-24T00:31:53.474 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-24T00:31:53.619 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T00:31:53.647 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-24T00:31:53.686 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-24T00:31:53.735 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-toml. 2024-04-24T00:31:53.752 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-24T00:31:53.793 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-24T00:31:54.049 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-test. 2024-04-24T00:31:54.077 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:54.105 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T00:31:54.115 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-test (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:54.133 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-24T00:31:54.171 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T00:31:54.483 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T00:31:54.500 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-24T00:31:54.532 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-24T00:31:54.793 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T00:31:54.807 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-24T00:31:54.851 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-24T00:31:55.564 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package radosgw. 2024-04-24T00:31:55.592 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../116-radosgw_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:55.630 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking radosgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:56.386 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T00:31:56.408 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T00:31:56.414 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2424-gb98aab6c-1jammy_all.deb ... 2024-04-24T00:31:56.425 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:56.452 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-volume (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:56.466 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking rbd-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:56.771 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-24T00:31:56.775 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T00:31:56.791 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:31:56.833 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libcephfs-dev (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:57.038 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T00:31:57.119 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-24T00:31:57.147 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-24T00:31:57.164 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T00:31:57.177 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T00:31:57.443 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-24T00:31:57.491 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-24T00:31:57.518 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-24T00:31:57.557 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-24T00:31:57.753 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-24T00:31:57.936 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T00:31:57.964 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-24T00:31:58.003 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T00:31:58.203 INFO:teuthology.orchestra.run.smithi064.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-24T00:31:58.266 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:31:58.433 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package pkg-config. 2024-04-24T00:31:58.461 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-24T00:31:58.492 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T00:31:58.736 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-24T00:31:58.764 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-24T00:31:58.774 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user cephadm....done 2024-04-24T00:31:58.803 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-24T00:31:58.905 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T00:31:59.173 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-24T00:31:59.223 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-24T00:31:59.251 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-24T00:31:59.289 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-24T00:31:59.424 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-24T00:31:59.541 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-24T00:31:59.801 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T00:31:59.817 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T00:31:59.845 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T00:31:59.884 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T00:32:00.077 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T00:32:00.203 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-24T00:32:00.204 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T00:32:00.231 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-24T00:32:00.270 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-24T00:32:00.513 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T00:32:00.557 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T00:32:00.584 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-24T00:32:00.623 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-24T00:32:00.890 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-24T00:32:00.968 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-py. 2024-04-24T00:32:00.996 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-24T00:32:01.035 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-24T00:32:01.124 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-24T00:32:01.338 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pygments. 2024-04-24T00:32:01.354 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-24T00:32:01.388 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T00:32:01.392 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-24T00:32:01.652 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-24T00:32:01.826 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T00:32:01.854 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-24T00:32:01.884 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-24T00:32:01.911 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-24T00:32:02.179 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-24T00:32:02.203 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-toml. 2024-04-24T00:32:02.231 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-24T00:32:02.270 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-24T00:32:02.447 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-argparse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:02.573 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T00:32:02.600 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-24T00:32:02.639 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T00:32:02.822 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T00:32:02.990 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T00:32:03.010 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T00:32:03.038 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-24T00:32:03.076 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-24T00:32:03.100 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T00:32:03.225 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-24T00:32:03.354 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T00:32:03.370 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-24T00:32:03.412 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-24T00:32:03.485 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-24T00:32:03.838 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-24T00:32:03.867 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-24T00:32:03.930 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T00:32:04.093 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package radosgw. 2024-04-24T00:32:04.097 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-24T00:32:04.119 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../116-radosgw_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:32:04.157 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking radosgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:04.457 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T00:32:04.566 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-24T00:32:04.834 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T00:32:04.961 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T00:32:04.977 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2424-gb98aab6c-1jammy_amd64.deb ... 2024-04-24T00:32:05.011 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking rbd-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:05.102 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-24T00:32:05.315 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-24T00:32:05.428 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up zip (3.0-12build2) ... 2024-04-24T00:32:05.554 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T00:32:05.566 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T00:32:05.675 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T00:32:05.905 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-24T00:32:06.199 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-24T00:32:06.272 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T00:32:06.515 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T00:32:06.624 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-24T00:32:06.642 INFO:teuthology.orchestra.run.smithi167.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-24T00:32:06.710 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:06.725 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T00:32:07.018 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-24T00:32:07.260 INFO:teuthology.orchestra.run.smithi167.stdout:Adding system user cephadm....done 2024-04-24T00:32:07.411 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-24T00:32:07.417 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T00:32:07.679 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T00:32:07.710 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-24T00:32:07.961 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-24T00:32:08.087 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-24T00:32:08.105 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T00:32:08.338 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T00:32:08.398 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T00:32:08.598 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T00:32:08.724 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-24T00:32:08.724 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-24T00:32:08.984 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-24T00:32:09.018 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T00:32:09.135 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:09.444 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-idna (3.3-1) ... 2024-04-24T00:32:09.520 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-24T00:32:09.763 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-24T00:32:09.779 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T00:32:10.039 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-24T00:32:10.291 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-24T00:32:10.542 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-24T00:32:10.793 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-24T00:32:11.034 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T00:32:11.044 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-ceph-argparse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:11.287 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T00:32:11.319 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:32:11.413 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T00:32:11.461 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-24T00:32:11.539 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T00:32:11.665 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-24T00:32:11.738 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T00:32:11.864 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-24T00:32:11.941 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-24T00:32:11.973 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T00:32:12.308 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-24T00:32:12.340 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-24T00:32:12.373 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-24T00:32:12.428 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T00:32:12.538 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-24T00:32:12.550 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-24T00:32:12.801 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:32:12.897 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T00:32:12.927 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-24T00:32:13.023 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-24T00:32:13.254 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-24T00:32:13.291 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T00:32:13.513 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-24T00:32:13.559 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-24T00:32:13.773 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-24T00:32:13.852 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up zip (3.0-12build2) ... 2024-04-24T00:32:13.953 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T00:32:14.015 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T00:32:14.516 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-24T00:32:14.588 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T00:32:14.834 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T00:32:14.839 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T00:32:14.960 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-24T00:32:14.969 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-24T00:32:15.252 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T00:32:15.387 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T00:32:15.408 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T00:32:15.685 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-24T00:32:15.713 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-24T00:32:16.022 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-24T00:32:16.044 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-24T00:32:16.273 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T00:32:16.287 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T00:32:16.374 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-24T00:32:16.500 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T00:32:16.589 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-24T00:32:16.655 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T00:32:16.852 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T00:32:16.940 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T00:32:16.978 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T00:32:17.283 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-24T00:32:17.379 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-24T00:32:17.509 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-24T00:32:17.611 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:17.638 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-24T00:32:17.748 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-24T00:32:17.895 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-idna (3.3-1) ... 2024-04-24T00:32:17.991 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:32:18.108 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T00:32:18.205 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T00:32:18.426 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T00:32:18.752 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T00:32:19.028 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-24T00:32:19.129 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-24T00:32:19.387 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T00:32:19.397 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-24T00:32:19.604 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:32:19.673 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T00:32:19.739 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-24T00:32:20.015 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T00:32:20.018 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-24T00:32:20.133 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-24T00:32:20.251 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T00:32:20.411 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T00:32:20.527 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-24T00:32:20.696 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T00:32:20.770 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-24T00:32:21.013 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:32:21.057 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T00:32:21.139 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-24T00:32:21.166 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librados2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:21.292 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:21.418 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-24T00:32:21.440 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-24T00:32:21.683 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-24T00:32:21.763 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:21.906 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libradosstriper1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:21.934 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-24T00:32:22.015 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-24T00:32:22.194 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T00:32:22.275 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librbd1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:22.401 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-modules-core (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:22.437 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-24T00:32:22.527 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:22.696 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T00:32:22.746 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T00:32:22.746 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T00:32:22.822 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-24T00:32:23.099 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T00:32:23.225 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T00:32:23.236 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs-dev (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:23.380 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rados (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:23.518 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-24T00:32:23.811 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-24T00:32:23.839 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librgw2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:23.966 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T00:32:24.045 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T00:32:24.174 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-24T00:32:24.297 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T00:32:24.386 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-24T00:32:24.632 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T00:32:24.758 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T00:32:25.143 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-24T00:32:25.277 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rbd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:25.395 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-24T00:32:25.403 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up rbd-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:25.504 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-24T00:32:25.554 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:25.680 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cephfs (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:25.780 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T00:32:25.798 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:25.906 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T00:32:26.257 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T00:32:26.290 INFO:teuthology.orchestra.run.smithi064.stdout:Adding group ceph....done 2024-04-24T00:32:26.567 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T00:32:26.684 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user ceph....done 2024-04-24T00:32:26.826 INFO:teuthology.orchestra.run.smithi064.stdout:Setting system user ceph properties....done 2024-04-24T00:32:26.835 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-24T00:32:26.846 INFO:teuthology.orchestra.run.smithi064.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T00:32:26.961 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-24T00:32:26.990 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T00:32:27.237 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-24T00:32:27.361 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T00:32:27.522 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T00:32:27.883 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-24T00:32:27.884 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-test (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:28.041 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up radosgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:28.277 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T00:32:28.519 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T00:32:28.520 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T00:32:28.570 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T00:32:28.906 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T00:32:28.976 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-base (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:29.032 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librados2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:29.158 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:29.284 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-24T00:32:29.376 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T00:32:29.554 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libcephfs2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:29.680 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libradosstriper1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:29.806 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-24T00:32:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mds (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:30.116 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librbd1 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:30.217 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-modules-core (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:30.240 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T00:32:30.240 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T00:32:30.343 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:30.555 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T00:32:30.556 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T00:32:30.683 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:30.891 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T00:32:30.892 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T00:32:31.011 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libcephfs-dev (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:31.129 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rados (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:31.255 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librgw2 (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:31.359 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-osd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:31.381 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T00:32:31.643 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T00:32:31.643 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T00:32:32.168 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:32.294 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:32.454 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mon (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:32.652 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rbd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:32.673 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T00:32:32.673 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T00:32:32.753 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up rbd-fuse (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:32.879 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:33.005 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cephfs (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:33.132 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-common (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:33.155 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:33.306 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:33.424 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-dashboard (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:33.550 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-volume (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:33.632 INFO:teuthology.orchestra.run.smithi167.stdout:Adding group ceph....done 2024-04-24T00:32:33.894 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T00:32:34.003 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T00:32:34.010 INFO:teuthology.orchestra.run.smithi167.stdout:Adding system user ceph....done 2024-04-24T00:32:34.156 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T00:32:34.178 INFO:teuthology.orchestra.run.smithi167.stdout:Setting system user ceph properties....done 2024-04-24T00:32:34.193 INFO:teuthology.orchestra.run.smithi167.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T00:32:34.333 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T00:32:34.636 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T00:32:35.083 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-test (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:35.210 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up radosgw (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:35.636 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T00:32:35.637 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T00:32:35.792 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T00:32:36.086 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-base (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:36.430 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T00:32:36.436 DEBUG:teuthology.parallel:result is None 2024-04-24T00:32:36.445 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T00:32:36.913 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mds (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:37.141 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T00:32:37.141 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T00:32:37.640 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:37.850 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T00:32:37.850 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T00:32:38.376 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-osd (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:38.636 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T00:32:38.636 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T00:32:39.119 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:39.245 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:39.396 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mon (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:39.606 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T00:32:39.606 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T00:32:40.073 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-cephadm (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:40.207 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:40.333 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-dashboard (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:40.476 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-volume (19.0.0-2424-gb98aab6c-1jammy) ... 2024-04-24T00:32:40.804 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T00:32:41.242 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T00:32:41.486 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T00:32:42.621 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T00:32:43.276 INFO:teuthology.orchestra.run.smithi167.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T00:32:43.281 DEBUG:teuthology.parallel:result is None 2024-04-24T00:32:43.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:32:43.422 DEBUG:teuthology.orchestra.run.smithi064:> dpkg-query -W -f '${Version}' ceph 2024-04-24T00:32:43.442 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:32:43.443 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:32:43.443 INFO:teuthology.task.install:The correct ceph version 19.0.0-2424-gb98aab6c-1jammy is installed. 2024-04-24T00:32:43.445 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-24T00:32:43.582 DEBUG:teuthology.orchestra.run.smithi167:> dpkg-query -W -f '${Version}' ceph 2024-04-24T00:32:43.601 INFO:teuthology.orchestra.run.smithi167.stdout:19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:32:43.601 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2424-gb98aab6c-1jammy 2024-04-24T00:32:43.601 INFO:teuthology.task.install:The correct ceph version 19.0.0-2424-gb98aab6c-1jammy is installed. 2024-04-24T00:32:43.603 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T00:32:43.603 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:43.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T00:32:43.618 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:32:43.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T00:32:43.655 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T00:32:43.656 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:43.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T00:32:43.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T00:32:43.729 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:32:43.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T00:32:43.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T00:32:43.795 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T00:32:43.795 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:43.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T00:32:43.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T00:32:43.866 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:32:43.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T00:32:43.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T00:32:43.931 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T00:32:43.931 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:43.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T00:32:43.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T00:32:44.001 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:32:44.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T00:32:44.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T00:32:44.068 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T00:32:44.210 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T00:32:44.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-04-24T00:32:44.213 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /var/log/ceph 2024-04-24T00:32:44.227 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T00:32:44.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T00:32:44.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T00:32:44.283 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T00:32:44.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T00:32:44.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T00:32:44.356 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-24T00:32:44.357 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T00:32:44.357 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533', 'branch': 'squid', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_deep_copy}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7670533', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9514, 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'sleep_before_teardown': 0, 'subset': '69/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b98aab6c00f483888c049d0d8e7bc507e115dd86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIM1uFhWanLvyIzTiPYVwYfZyZvMwYROwt4gjDdoui72/apTLiOuZUn4tfs/iRVmS2c3b1yqfEiPs0RUrelx7yE=', 'smithi167.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPMmKtpw4oM6/fecT9g5lapXKGYUK7S9AQfIwQYxCtfbqmnCmgSfN5b+su8ADMcyUnCsfH0VtgDhTDW4L+7Rn44='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'deep_copy': True, 'ops': 6000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-23_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T00:32:44.357 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T00:32:44.383 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T00:32:44.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T00:32:44.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T00:32:44.444 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:44.444 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T00:32:44.490 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T00:32:44.490 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-04-24T00:32:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T00:32:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:32:44.539 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 878 Links: 1 2024-04-24T00:32:44.539 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:32:44.539 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-24 00:27:18.687902506 +0000 2024-04-24T00:32:44.539 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-24 00:27:18.431904792 +0000 2024-04-24T00:32:44.539 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-24 00:27:18.431904792 +0000 2024-04-24T00:32:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2024-04-24T00:32:44.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T00:32:44.593 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-24T00:32:44.593 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-24T00:32:44.593 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.0004459 s, 1.1 MB/s 2024-04-24T00:32:44.595 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T00:32:44.644 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-04-24T00:32:44.690 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T00:32:44.691 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:32:44.691 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-04-24T00:32:44.691 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:32:44.691 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-24 00:27:18.911900505 +0000 2024-04-24T00:32:44.691 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-24 00:27:18.907900541 +0000 2024-04-24T00:32:44.691 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-24 00:27:18.907900541 +0000 2024-04-24T00:32:44.691 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2024-04-24T00:32:44.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T00:32:44.745 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-24T00:32:44.746 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-24T00:32:44.746 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000392223 s, 1.3 MB/s 2024-04-24T00:32:44.747 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T00:32:44.796 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-04-24T00:32:44.842 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T00:32:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:32:44.843 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-04-24T00:32:44.843 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:32:44.843 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-24 00:27:19.695893502 +0000 2024-04-24T00:32:44.843 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-24 00:27:19.451895681 +0000 2024-04-24T00:32:44.843 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-24 00:27:19.451895681 +0000 2024-04-24T00:32:44.843 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2024-04-24T00:32:44.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T00:32:44.897 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-24T00:32:44.897 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-24T00:32:44.897 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000382128 s, 1.3 MB/s 2024-04-24T00:32:44.898 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T00:32:44.947 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-04-24T00:32:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T00:32:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:32:44.994 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 932 Links: 1 2024-04-24T00:32:44.995 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:32:44.995 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-24 00:27:20.219888822 +0000 2024-04-24T00:32:44.995 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-24 00:27:19.963891108 +0000 2024-04-24T00:32:44.995 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-24 00:27:19.963891108 +0000 2024-04-24T00:32:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2024-04-24T00:32:44.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T00:32:45.049 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-24T00:32:45.049 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-24T00:32:45.049 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000376523 s, 1.4 MB/s 2024-04-24T00:32:45.050 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T00:32:45.100 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T00:32:45.100 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:32:45.100 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T00:32:45.106 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T00:32:45.106 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2024-04-24T00:32:45.153 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T00:32:45.153 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:32:45.153 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 878 Links: 1 2024-04-24T00:32:45.153 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:32:45.153 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-24 00:27:18.223616641 +0000 2024-04-24T00:32:45.153 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-24 00:27:17.939619178 +0000 2024-04-24T00:32:45.153 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-24 00:27:17.939619178 +0000 2024-04-24T00:32:45.153 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2024-04-24T00:32:45.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T00:32:45.207 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-24T00:32:45.207 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-24T00:32:45.208 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000438939 s, 1.2 MB/s 2024-04-24T00:32:45.209 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T00:32:45.259 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2024-04-24T00:32:45.304 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T00:32:45.304 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:32:45.305 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-04-24T00:32:45.305 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:32:45.305 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-24 00:27:18.691612461 +0000 2024-04-24T00:32:45.305 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-24 00:27:18.455614569 +0000 2024-04-24T00:32:45.305 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-24 00:27:18.455614569 +0000 2024-04-24T00:32:45.305 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2024-04-24T00:32:45.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T00:32:45.358 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-24T00:32:45.358 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-24T00:32:45.358 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000312367 s, 1.6 MB/s 2024-04-24T00:32:45.359 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T00:32:45.405 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2024-04-24T00:32:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T00:32:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:32:45.453 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 907 Links: 1 2024-04-24T00:32:45.453 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:32:45.453 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-24 00:27:19.175608138 +0000 2024-04-24T00:32:45.453 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-24 00:27:18.915610460 +0000 2024-04-24T00:32:45.453 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-24 00:27:18.915610460 +0000 2024-04-24T00:32:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2024-04-24T00:32:45.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T00:32:45.507 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-24T00:32:45.507 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-24T00:32:45.507 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000419307 s, 1.2 MB/s 2024-04-24T00:32:45.508 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T00:32:45.557 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2024-04-24T00:32:45.605 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T00:32:45.605 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T00:32:45.605 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 932 Links: 1 2024-04-24T00:32:45.605 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T00:32:45.605 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-04-24 00:27:19.679603635 +0000 2024-04-24T00:32:45.605 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-04-24 00:27:19.415605994 +0000 2024-04-24T00:32:45.605 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-04-24 00:27:19.415605994 +0000 2024-04-24T00:32:45.605 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2024-04-24T00:32:45.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T00:32:45.659 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-04-24T00:32:45.659 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-04-24T00:32:45.659 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000349391 s, 1.5 MB/s 2024-04-24T00:32:45.660 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T00:32:45.710 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T00:32:45.710 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-24T00:32:45.710 INFO:tasks.ceph:Generating config... 2024-04-24T00:32:45.711 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T00:32:45.711 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T00:32:45.711 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T00:32:45.711 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T00:32:45.711 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T00:32:45.712 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T00:32:45.712 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T00:32:45.712 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T00:32:45.712 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T00:32:45.713 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-24T00:32:45.713 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T00:32:45.713 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-24T00:32:45.713 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T00:32:45.713 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-24T00:32:45.713 INFO:tasks.ceph:Setting up mon.a... 2024-04-24T00:32:45.713 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-24T00:32:45.748 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2024-04-24T00:32:45.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-24T00:32:45.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T00:32:45.894 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.64'), ('mon.c', '[v2:172.21.15.64:3301,v1:172.21.15.64:6790]'), ('mon.b', '172.21.15.167')] 2024-04-24T00:32:45.895 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],172.21.15.167', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-24T00:32:45.895 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:45.895 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T00:32:45.942 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.64 --addv c '[v2:172.21.15.64:3301,v1:172.21.15.64:6790]' --add b 172.21.15.167 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:32:46.030 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid 6a70d87d-4453-47be-af2a-5590f9851254 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 6a70d87d-4453-47be-af2a-5590f9851254 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-04-24T00:32:46.029181+0000 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-04-24T00:32:46.029181+0000 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 1 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-04-24T00:32:46.031 INFO:teuthology.orchestra.run.smithi064.stdout:2: [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon.c 2024-04-24T00:32:46.032 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-24T00:32:46.033 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T00:32:46.037 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6a70d87d-4453-47be-af2a-5590f9851254... 2024-04-24T00:32:46.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T00:32:46.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T00:32:46.112 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-04-24T00:32:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-04-24T00:32:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T00:32:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-04-24T00:32:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-04-24T00:32:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-04-24T00:32:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-04-24T00:32:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-04-24T00:32:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-04-24T00:32:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-04-24T00:32:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T00:32:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-04-24T00:32:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T00:32:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T00:32:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-04-24T00:32:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-04-24T00:32:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-04-24T00:32:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-04-24T00:32:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-04-24T00:32:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-04-24T00:32:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-04-24T00:32:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-04-24T00:32:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T00:32:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T00:32:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-04-24T00:32:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T00:32:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-04-24T00:32:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-04-24T00:32:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-04-24T00:32:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-04-24T00:32:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-04-24T00:32:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],172.21.15.167" 2024-04-24T00:32:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-04-24T00:32:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on pool no app = False 2024-04-24T00:32:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-04-24T00:32:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 6a70d87d-4453-47be-af2a-5590f9851254 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-04-24T00:32:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = zlib 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 1/20 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 1/20 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-04-24T00:32:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 4/10 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: osd blocked scrub grace period = 3600 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: osd max markdown count = 1000 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-04-24T00:32:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-04-24T00:32:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-04-24T00:32:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T00:32:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T00:32:46.126 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-04-24T00:32:46.126 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-04-24T00:32:46.126 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-04-24T00:32:46.127 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: chdir = "" 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: auth supported = cephx 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: filestore xattr use omap = true 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: mon clock drift allowed = 1.000 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: osd crush chooseleaf type = 0 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: auth debug = true 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on old message = true 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on bug = true 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: mon pg warn max object skew = 0 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default size = 2 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow primary affinity = true 2024-04-24T00:32:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow pg remap = true 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on legacy crush tunables = false 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on crush straw calc version zero = false 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on no sortbitwise = false 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on osd down out interval zero = false 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on too few osds = false 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: mon_allow_pool_size_one = true 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: osd default data pool replay window = 5 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: mon allow pool delete = true 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: mon cluster log file level = debug 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: debug asserts on shutdown = true 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: mon health detail to clog = false 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: mon host = "172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],172.21.15.167" 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: mon client directed command retry = 5 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on pool no app = False 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: ms inject socket failures = 5000 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = 6a70d87d-4453-47be-af2a-5590f9851254 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.131 INFO:teuthology.orchestra.run.smithi167.stdout:[osd] 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd journal size = 100 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub load threshold = 5.0 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub max interval = 600 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock profile = high_recovery_ops 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd recover clone overlap = true 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd recovery max chunk = 1048576 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug shutdown = true 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug op order = true 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify stray on activate = true 2024-04-24T00:32:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug trim objects = true 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: osd open classes on start = true 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug pg log writeout = true 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: osd deep scrub update digest min age = 30 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: osd map max advance = 10 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: journal zero on create = true 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: filestore ondisk finisher threads = 3 2024-04-24T00:32:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: filestore apply finisher threads = 3 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: bdev debug aio = true 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug misdirected ops = true 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore block size = 96636764160 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore compression algorithm = zlib 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore compression mode = aggressive 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore fsck on mount = True 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluefs = 1/20 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluestore = 1/20 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-24T00:32:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: debug osd = 20 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: debug rocksdb = 4/10 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd full ratio = 0.9 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd nearfull ratio = 0.8 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: osd blocked scrub grace period = 3600 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: osd failsafe full ratio = 0.95 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: osd max markdown count = 1000 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: osd objectstore = bluestore 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: osd shutdown pgref assert = True 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr] 2024-04-24T00:32:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: debug mgr = 20 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: mgr/telemetry/nag = false 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout:[mon] 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: debug paxos = 20 2024-04-24T00:32:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: mon data avail warn = 5 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: mon mgr mkfs grace = 240 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd reporter subtree level = osd 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd prime pg temp = true 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T00:32:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: # 1m isn't quite enough 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: mon_down_mkfs_grace = 2m 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_filestore_osds = false 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout:[client] 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: rgw cache enabled = true 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable ops log = true 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable usage log = true 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T00:32:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T00:32:46.139 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.a] 2024-04-24T00:32:46.139 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.c] 2024-04-24T00:32:46.139 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.b] 2024-04-24T00:32:46.139 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-24T00:32:46.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-24T00:32:46.217 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T00:32:46.217 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:46.217 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-24T00:32:46.222 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:46.222 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-24T00:32:46.269 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:32:46.269 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:46.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T00:32:46.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T00:32:46.333 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:32:46.333 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:32:46.382 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:32:46.382 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:32:46.382 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T00:32:46.382 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T00:32:46.405 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:32:46.405 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:32:46.453 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T00:32:46.453 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T00:32:46.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-24T00:32:46.497 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-24T00:32:46.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T00:32:46.542 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T00:32:46.551 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T00:32:46.551 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T00:32:46.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T00:32:46.587 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-24T00:32:46.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-24T00:32:46.634 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-24T00:32:46.643 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T00:32:46.644 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-24T00:32:46.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-24T00:32:46.657 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T00:32:46.657 INFO:tasks.ceph:role: osd.0 2024-04-24T00:32:46.658 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:32:46.658 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T00:32:46.722 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:32:46.722 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:32:46.722 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:32:46.723 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:32:46.723 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:32:46.723 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:32:46.723 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:32:46.723 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:32:46.723 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:32:46.723 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:32:47.211 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-04-24T00:32:47.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-04-24T00:32:47.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-24T00:32:47.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-24T00:32:47.391 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:32:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:32:47.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-24T00:32:47.446 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T00:32:47.446 INFO:tasks.ceph:role: osd.1 2024-04-24T00:32:47.446 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:32:47.447 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T00:32:47.509 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:32:47.510 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:32:47.947 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-04-24T00:32:47.949 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-04-24T00:32:47.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-24T00:32:48.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-24T00:32:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:32:48.091 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:32:48.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-24T00:32:48.146 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T00:32:48.146 INFO:tasks.ceph:role: osd.2 2024-04-24T00:32:48.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:32:48.146 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T00:32:48.209 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:32:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:32:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:32:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:32:48.210 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:32:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:32:48.210 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:32:48.211 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:32:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:32:48.211 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:32:48.667 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-04-24T00:32:48.669 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-04-24T00:32:48.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-24T00:32:48.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-24T00:32:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:32:48.778 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:32:48.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-24T00:32:48.833 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T00:32:48.833 INFO:tasks.ceph:role: osd.3 2024-04-24T00:32:48.833 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:32:48.834 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:32:48.898 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:32:49.395 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-04-24T00:32:49.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-04-24T00:32:49.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-24T00:32:49.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-24T00:32:49.563 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:32:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:32:49.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:32:49.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:49.669+0000 7f176d9177c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-24T00:32:49.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:49.673+0000 7f176d9177c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-24T00:32:49.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:49.673+0000 7f176d9177c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T00:32:49.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:49.673+0000 7f176d9177c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T00:32:49.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:49.673+0000 7f176d9177c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T00:32:49.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:49.685+0000 7f176d9177c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-24T00:32:52.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T00:32:52.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:32:52.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:52.769+0000 7f590ea677c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-24T00:32:52.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:52.769+0000 7f590ea677c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-24T00:32:52.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:52.769+0000 7f590ea677c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T00:32:52.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:52.769+0000 7f590ea677c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T00:32:52.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:52.769+0000 7f590ea677c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T00:32:52.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:52.773+0000 7f590ea677c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-24T00:32:55.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T00:32:55.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:32:55.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:55.901+0000 7ffa639ee7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-24T00:32:55.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:55.901+0000 7ffa639ee7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-24T00:32:55.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:55.901+0000 7ffa639ee7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T00:32:55.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:55.901+0000 7ffa639ee7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T00:32:55.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:55.901+0000 7ffa639ee7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T00:32:55.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:55.901+0000 7ffa639ee7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-24T00:32:58.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T00:32:58.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:32:59.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:59.009+0000 7f1339f0e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-24T00:32:59.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:59.009+0000 7f1339f0e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-24T00:32:59.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:59.009+0000 7f1339f0e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T00:32:59.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:59.009+0000 7f1339f0e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T00:32:59.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:59.009+0000 7f1339f0e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T00:32:59.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:32:59.009+0000 7f1339f0e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-24T00:33:02.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T00:33:02.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-24T00:33:02.081 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T00:33:02.081 INFO:tasks.ceph:role: osd.4 2024-04-24T00:33:02.081 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:33:02.082 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:33:02.143 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:33:02.607 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-24T00:33:02.610 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-24T00:33:02.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-24T00:33:02.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-24T00:33:02.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:33:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:33:02.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-24T00:33:02.812 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T00:33:02.812 INFO:tasks.ceph:role: osd.5 2024-04-24T00:33:02.812 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:33:02.812 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T00:33:02.878 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:33:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:33:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:33:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:33:02.878 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:33:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:33:02.879 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:33:02.879 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:33:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:33:02.879 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:33:03.310 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-24T00:33:03.313 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-24T00:33:03.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-24T00:33:03.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-24T00:33:03.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:33:03.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:33:03.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-24T00:33:03.479 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T00:33:03.479 INFO:tasks.ceph:role: osd.6 2024-04-24T00:33:03.479 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:33:03.479 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T00:33:03.539 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:33:03.540 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:33:04.008 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-24T00:33:04.010 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-24T00:33:04.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-24T00:33:04.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-24T00:33:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:33:04.150 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:33:04.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-24T00:33:04.203 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T00:33:04.203 INFO:tasks.ceph:role: osd.7 2024-04-24T00:33:04.203 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:33:04.203 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T00:33:04.269 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T00:33:04.744 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-04-24T00:33:04.746 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-04-24T00:33:04.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-24T00:33:04.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-24T00:33:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:33:04.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T00:33:04.892 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:33:05.006 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:05.001+0000 7f8ee64bc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-24T00:33:05.010 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:05.005+0000 7f8ee64bc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-24T00:33:05.010 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:05.005+0000 7f8ee64bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T00:33:05.010 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:05.005+0000 7f8ee64bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T00:33:05.010 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:05.005+0000 7f8ee64bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T00:33:05.015 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:05.009+0000 7f8ee64bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-24T00:33:07.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T00:33:08.002 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:33:08.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:08.077+0000 7fcc343627c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-24T00:33:08.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:08.077+0000 7fcc343627c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-24T00:33:08.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:08.077+0000 7fcc343627c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T00:33:08.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:08.077+0000 7fcc343627c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T00:33:08.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:08.077+0000 7fcc343627c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T00:33:08.084 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:08.081+0000 7fcc343627c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-24T00:33:11.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T00:33:11.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:33:11.219 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:11.212+0000 7efeedc7f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-24T00:33:11.220 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:11.216+0000 7efeedc7f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-24T00:33:11.220 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:11.216+0000 7efeedc7f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T00:33:11.220 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:11.216+0000 7efeedc7f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T00:33:11.220 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:11.216+0000 7efeedc7f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T00:33:11.220 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:11.216+0000 7efeedc7f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-24T00:33:14.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-24T00:33:14.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:33:14.342 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:14.336+0000 7f62d073a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-24T00:33:14.343 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:14.336+0000 7f62d073a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-24T00:33:14.343 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:14.336+0000 7f62d073a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T00:33:14.343 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:14.336+0000 7f62d073a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T00:33:14.343 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:14.336+0000 7f62d073a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T00:33:14.343 INFO:teuthology.orchestra.run.smithi167.stderr:2024-04-24T00:33:14.336+0000 7f62d073a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-24T00:33:17.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-24T00:33:17.393 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T00:33:17.393 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:17.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-24T00:33:17.409 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:17.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-24T00:33:17.462 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:17.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-24T00:33:17.517 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:17.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-24T00:33:17.573 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:17.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-24T00:33:17.629 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:17.629 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-24T00:33:17.643 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:17.643 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-24T00:33:17.699 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:17.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-24T00:33:17.755 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:17.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-24T00:33:17.811 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:17.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-24T00:33:17.867 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:17.867 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-24T00:33:17.873 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:17.873 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-24T00:33:17.916 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T00:33:17.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T00:33:17.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T00:33:17.962 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2024-04-24T00:33:17.962 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAuUyhmim2SHRAAvYct0RQH9t+OzIY9JDaLhA== 2024-04-24T00:33:17.962 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-04-24T00:33:17.962 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAxUyhm235IKBAA2L+/TFC3EpPyWnpy7Nn3Jg== 2024-04-24T00:33:17.962 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-04-24T00:33:17.965 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA0Uyhmt448LhAA6UXn826zjydsamMZ6o9dpQ== 2024-04-24T00:33:17.965 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA3UyhmV/LtNRAAYKaV38RtKd05lhNiHOV7tQ== 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA7UyhmATDBABAAo120w9pKV8gdBWPqpJytbw== 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAuUyhmHvIlIBAABbSJ2ZVryg8Odi6U7ZalSQ== 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBBUyhmPp40ABAAoLsLkMG05j0szwziAq5MPA== 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBEUyhm0lHFBBAAzMTT+IAqXnlLRi4Y2LABGQ== 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBHUyhmnOjsDBAA5/ESljwSmdkUO72ImQCkWw== 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBKUyhm8MVAFBAAdl24jvywAfE6dJniMmCLRg== 2024-04-24T00:33:17.966 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-04-24T00:33:17.967 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAuUyhm3pruIhAA59HU9pUNjIKHRNbWpGLO3w== 2024-04-24T00:33:17.967 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-04-24T00:33:17.967 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAuUyhmd0icJRAAkHjNYSbjEhu19ch+wm7UpQ== 2024-04-24T00:33:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.y] 2024-04-24T00:33:17.970 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQAuUyhmim2SHRAAvYct0RQH9t+OzIY9JDaLhA== 2024-04-24T00:33:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.0] 2024-04-24T00:33:17.970 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQAxUyhm235IKBAA2L+/TFC3EpPyWnpy7Nn3Jg== 2024-04-24T00:33:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.1] 2024-04-24T00:33:17.970 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA0Uyhmt448LhAA6UXn826zjydsamMZ6o9dpQ== 2024-04-24T00:33:17.970 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.2] 2024-04-24T00:33:17.970 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA3UyhmV/LtNRAAYKaV38RtKd05lhNiHOV7tQ== 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.3] 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA7UyhmATDBABAAo120w9pKV8gdBWPqpJytbw== 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.x] 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQAuUyhmHvIlIBAABbSJ2ZVryg8Odi6U7ZalSQ== 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.4] 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBBUyhmPp40ABAAoLsLkMG05j0szwziAq5MPA== 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.5] 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBEUyhm0lHFBBAAzMTT+IAqXnlLRi4Y2LABGQ== 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.6] 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBHUyhmnOjsDBAA5/ESljwSmdkUO72ImQCkWw== 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.7] 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBKUyhm8MVAFBAAdl24jvywAfE6dJniMmCLRg== 2024-04-24T00:33:17.971 INFO:teuthology.orchestra.run.smithi167.stdout:[client.0] 2024-04-24T00:33:17.972 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQAuUyhm3pruIhAA59HU9pUNjIKHRNbWpGLO3w== 2024-04-24T00:33:17.972 INFO:teuthology.orchestra.run.smithi167.stdout:[client.1] 2024-04-24T00:33:17.972 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQAuUyhmd0icJRAAkHjNYSbjEhu19ch+wm7UpQ== 2024-04-24T00:33:17.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T00:33:18.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T00:33:18.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.146 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.231 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.318 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T00:33:18.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T00:33:18.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T00:33:18.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:33:18.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:33:18.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:33:18.897 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T00:33:18.941 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T00:33:18.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-24T00:33:18.994 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T00:33:19.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-24T00:33:19.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-24T00:33:19.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T00:33:19.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-24T00:33:19.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-24T00:33:19.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T00:33:19.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-24T00:33:19.382 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:33:19.384 DEBUG:teuthology.orchestra.run.smithi167:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T00:33:19.431 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-24T00:33:19.433 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-24T00:33:19.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-24T00:33:19.435 INFO:tasks.ceph.mon.a:Started 2024-04-24T00:33:19.435 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-24T00:33:19.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-24T00:33:19.438 INFO:tasks.ceph.mon.c:Started 2024-04-24T00:33:19.438 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-24T00:33:19.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-24T00:33:19.476 INFO:tasks.ceph.mon.b:Started 2024-04-24T00:33:19.476 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-24T00:33:19.477 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-24T00:33:19.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-24T00:33:19.479 INFO:tasks.ceph.mgr.y:Started 2024-04-24T00:33:19.479 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-24T00:33:19.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-24T00:33:19.481 INFO:tasks.ceph.mgr.x:Started 2024-04-24T00:33:19.482 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T00:33:19.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2024-04-24T00:33:25.124 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:25.116+0000 7f09128a5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T00:33:25.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:25.156+0000 7f443e824180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T00:33:25.650 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:25.644+0000 7f09128a5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T00:33:25.681 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:25.676+0000 7f443e824180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T00:33:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-04-24T00:33:25.726 INFO:tasks.ceph:check_enable_crimson: False 2024-04-24T00:33:25.726 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-24T00:33:25.726 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:25.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-24T00:33:25.739 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:25.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-24T00:33:25.791 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:25.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-24T00:33:25.844 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-24T00:33:25.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-24T00:33:25.878 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:25.872+0000 7f09128a5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T00:33:25.890 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:25.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-24T00:33:25.891 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:25.884+0000 7f443e824180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T00:33:25.903 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:25.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-24T00:33:25.959 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:25.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-24T00:33:26.011 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-04-24T00:33:26.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-24T00:33:26.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 1c80b3f4-99ee-4725-90a8-920116161bf5 0 2024-04-24T00:33:26.316 INFO:teuthology.orchestra.run.smithi167.stdout:0 2024-04-24T00:33:26.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 20e1331a-f8a2-48fd-b2b8-b7a92f21d78e 1 2024-04-24T00:33:26.443 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:26.436+0000 7f09128a5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T00:33:26.475 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:26.468+0000 7f443e824180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T00:33:26.499 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:26.492+0000 7f09128a5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T00:33:26.527 INFO:teuthology.orchestra.run.smithi167.stdout:1 2024-04-24T00:33:26.529 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:26.524+0000 7f443e824180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T00:33:26.544 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new a51557a0-9634-4a20-82b0-b54cdd36b166 2 2024-04-24T00:33:26.555 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:26.548+0000 7f09128a5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T00:33:26.585 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:26.580+0000 7f443e824180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T00:33:26.635 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:26.628+0000 7f09128a5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T00:33:26.665 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:26.660+0000 7f443e824180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T00:33:26.690 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:26.684+0000 7f09128a5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T00:33:26.717 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:26.712+0000 7f443e824180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T00:33:26.737 INFO:teuthology.orchestra.run.smithi167.stdout:2 2024-04-24T00:33:26.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new aafffa79-f532-4441-b1d6-f6441c5eebf8 3 2024-04-24T00:33:26.772 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:26.764+0000 7f09128a5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T00:33:26.797 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:26.792+0000 7f443e824180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T00:33:26.826 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:26.820+0000 7f09128a5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T00:33:26.852 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:26.848+0000 7f443e824180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T00:33:26.984 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:26.976+0000 7f09128a5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T00:33:26.986 INFO:teuthology.orchestra.run.smithi167.stdout:3 2024-04-24T00:33:27.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 78f05556-598e-4be2-b682-e3f58863eb82 4 2024-04-24T00:33:27.008 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:27.004+0000 7f443e824180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T00:33:27.048 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:27.040+0000 7f09128a5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T00:33:27.072 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:27.068+0000 7f443e824180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T00:33:27.193 INFO:teuthology.orchestra.run.smithi167.stdout:4 2024-04-24T00:33:27.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5 5 2024-04-24T00:33:27.447 INFO:teuthology.orchestra.run.smithi167.stdout:5 2024-04-24T00:33:27.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 6dad12bf-94de-482a-a402-49e67d5e5dc7 6 2024-04-24T00:33:27.661 INFO:teuthology.orchestra.run.smithi167.stdout:6 2024-04-24T00:33:27.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new e6734cfe-3c24-4ddd-8ef0-5ed450efe772 7 2024-04-24T00:33:27.685 INFO:tasks.ceph.mgr.y.smithi064.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-24T00:33:27.685 INFO:tasks.ceph.mgr.y.smithi064.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-24T00:33:27.685 INFO:tasks.ceph.mgr.y.smithi064.stderr: from numpy import show_config as show_numpy_config 2024-04-24T00:33:27.718 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:27.712+0000 7f09128a5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T00:33:27.719 INFO:tasks.ceph.mgr.x.smithi167.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-24T00:33:27.719 INFO:tasks.ceph.mgr.x.smithi167.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-24T00:33:27.719 INFO:tasks.ceph.mgr.x.smithi167.stderr: from numpy import show_config as show_numpy_config 2024-04-24T00:33:27.764 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:27.760+0000 7f443e824180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T00:33:27.860 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:27.852+0000 7f09128a5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T00:33:27.866 INFO:teuthology.orchestra.run.smithi167.stdout:7 2024-04-24T00:33:27.883 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-24T00:33:27.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-24T00:33:27.885 INFO:tasks.ceph.osd.0:Started 2024-04-24T00:33:27.886 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-24T00:33:27.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-24T00:33:27.888 INFO:tasks.ceph.osd.1:Started 2024-04-24T00:33:27.888 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-24T00:33:27.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-24T00:33:27.890 INFO:tasks.ceph.osd.2:Started 2024-04-24T00:33:27.891 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-24T00:33:27.891 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-24T00:33:27.893 INFO:tasks.ceph.osd.3:Started 2024-04-24T00:33:27.893 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-24T00:33:27.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-24T00:33:27.905 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:27.900+0000 7f443e824180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T00:33:27.906 INFO:tasks.ceph.osd.4:Started 2024-04-24T00:33:27.906 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-24T00:33:27.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-24T00:33:27.908 INFO:tasks.ceph.osd.5:Started 2024-04-24T00:33:27.908 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-24T00:33:27.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-24T00:33:27.910 INFO:tasks.ceph.osd.6:Started 2024-04-24T00:33:27.911 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-24T00:33:27.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-24T00:33:27.912 INFO:tasks.ceph.osd.7:Started 2024-04-24T00:33:27.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T00:33:28.067 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:28.060+0000 7f09128a5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T00:33:28.089 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:28.084+0000 7f443e824180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T00:33:28.112 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:28.112 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:33:27.862090+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:33:27.862090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:33:28.129 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-24T00:33:28.129 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T00:33:28.134 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:28.128+0000 7f09128a5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T00:33:28.152 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:28.148+0000 7f443e824180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T00:33:28.275 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:28.268+0000 7f09128a5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T00:33:28.289 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:28.284+0000 7f443e824180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T00:33:28.523 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:28.516+0000 7f82b75277c0 -1 Falling back to public interface 2024-04-24T00:33:28.525 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:28.520+0000 7f5c728787c0 -1 Falling back to public interface 2024-04-24T00:33:28.527 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:28.520+0000 7f36aea087c0 -1 Falling back to public interface 2024-04-24T00:33:28.528 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:28.524+0000 7f185b1927c0 -1 Falling back to public interface 2024-04-24T00:33:28.533 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:28.528+0000 7f025b2d97c0 -1 Falling back to public interface 2024-04-24T00:33:28.536 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:28.532+0000 7fd7a93cd7c0 -1 Falling back to public interface 2024-04-24T00:33:28.536 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:28.532+0000 7f2faa4e87c0 -1 Falling back to public interface 2024-04-24T00:33:28.541 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:28.536+0000 7f63e71d07c0 -1 Falling back to public interface 2024-04-24T00:33:28.908 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:28.900+0000 7f09128a5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T00:33:28.934 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T00:33:28.936 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:28.932+0000 7f443e824180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T00:33:28.987 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:28.980+0000 7f09128a5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T00:33:29.010 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:29.004+0000 7f443e824180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T00:33:29.116 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-24T00:33:29.116 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:33:27.862090+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:33:27.862090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:33:29.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-24T00:33:29.181 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:29.172+0000 7f09128a5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T00:33:29.192 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:29.188+0000 7f443e824180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T00:33:29.257 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:29.252+0000 7f09128a5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T00:33:29.265 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:29.260+0000 7f443e824180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T00:33:29.553 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:29.548+0000 7f443e824180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T00:33:29.559 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:29.552+0000 7f09128a5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T00:33:29.614 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:29.608+0000 7f443e824180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T00:33:29.624 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-04-24T00:33:29.616+0000 7f09128a5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T00:33:30.658 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:30.652+0000 7f43e5c71640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T00:33:30.660 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:33:30.656+0000 7f43e5c71640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T00:33:32.294 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:32.288+0000 7f82b75277c0 -1 osd.3 0 log_to_monitors true 2024-04-24T00:33:32.372 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:32.368+0000 7f185b1927c0 -1 osd.5 0 log_to_monitors true 2024-04-24T00:33:32.386 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:32.380+0000 7f2faa4e87c0 -1 osd.4 0 log_to_monitors true 2024-04-24T00:33:32.416 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:32.408+0000 7f36aea087c0 -1 osd.2 0 log_to_monitors true 2024-04-24T00:33:32.454 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:32.448+0000 7f5c728787c0 -1 osd.6 0 log_to_monitors true 2024-04-24T00:33:32.457 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:32.452+0000 7fd7a93cd7c0 -1 osd.7 0 log_to_monitors true 2024-04-24T00:33:32.470 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:32.464+0000 7f025b2d97c0 -1 osd.0 0 log_to_monitors true 2024-04-24T00:33:32.479 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:32.472+0000 7f63e71d07c0 -1 osd.1 0 log_to_monitors true 2024-04-24T00:33:35.938 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T00:33:36.290 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-24T00:33:36.290 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":12,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:33:33.659650+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T00:33:27.862090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:33:36.314 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-24T00:33:36.865 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:36.860+0000 7f5c6e821640 -1 osd.6 0 waiting for initial osdmap 2024-04-24T00:33:36.879 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:36.872+0000 7fd7a5376640 -1 osd.7 0 waiting for initial osdmap 2024-04-24T00:33:36.892 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:36.888+0000 7f5c66656640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:36.912 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:36.908+0000 7fd79d1ab640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:36.965 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:36.960+0000 7f2fa6491640 -1 osd.4 0 waiting for initial osdmap 2024-04-24T00:33:36.979 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:36.976+0000 7f185794d640 -1 osd.5 0 waiting for initial osdmap 2024-04-24T00:33:36.981 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:36.976+0000 7f63e398b640 -1 osd.1 0 waiting for initial osdmap 2024-04-24T00:33:36.992 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:36.988+0000 7f2f9e2c6640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:37.001 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:36.996+0000 7f0257282640 -1 osd.0 0 waiting for initial osdmap 2024-04-24T00:33:37.005 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:37.000+0000 7f63dafae640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:37.007 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:37.004+0000 7f184ef70640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:37.034 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:37.028+0000 7f024f0b7640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:37.056 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:37.048+0000 7f36aa9b1640 -1 osd.2 0 waiting for initial osdmap 2024-04-24T00:33:37.081 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:37.076+0000 7f36a27e6640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:37.175 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:37.168+0000 7f82b34d0640 -1 osd.3 0 waiting for initial osdmap 2024-04-24T00:33:37.194 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:37.188+0000 7f82ab305640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:37.900 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:37.896+0000 7f5c6215d640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:33:43.122 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T00:33:43.430 INFO:teuthology.misc.health.smithi064.stdout: 2024-04-24T00:33:43.430 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":17,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:33:41.921231+0000","last_up_change":"2024-04-24T00:33:37.880801+0000","last_in_change":"2024-04-24T00:33:27.862090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:33:39.734814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6817","nonce":3307242121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6819","nonce":3307242121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6823","nonce":3307242121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6821","nonce":3307242121}]},"public_addr":"172.21.15.64:6817/3307242121","cluster_addr":"172.21.15.64:6819/3307242121","heartbeat_back_addr":"172.21.15.64:6823/3307242121","heartbeat_front_addr":"172.21.15.64:6821/3307242121","state":["exists","up"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6825","nonce":637720411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6827","nonce":637720411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6831","nonce":637720411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6829","nonce":637720411}]},"public_addr":"172.21.15.64:6825/637720411","cluster_addr":"172.21.15.64:6827/637720411","heartbeat_back_addr":"172.21.15.64:6831/637720411","heartbeat_front_addr":"172.21.15.64:6829/637720411","state":["exists","up"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6809","nonce":820024667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6811","nonce":820024667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6815","nonce":820024667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6813","nonce":820024667}]},"public_addr":"172.21.15.64:6809/820024667","cluster_addr":"172.21.15.64:6811/820024667","heartbeat_back_addr":"172.21.15.64:6815/820024667","heartbeat_front_addr":"172.21.15.64:6813/820024667","state":["exists","up"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6801","nonce":1974534323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6803","nonce":1974534323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6807","nonce":1974534323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6805","nonce":1974534323}]},"public_addr":"172.21.15.64:6801/1974534323","cluster_addr":"172.21.15.64:6803/1974534323","heartbeat_back_addr":"172.21.15.64:6807/1974534323","heartbeat_front_addr":"172.21.15.64:6805/1974534323","state":["exists","up"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6818","nonce":1642543604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6821","nonce":1642543604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6828","nonce":1642543604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6823","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6824","nonce":1642543604}]},"public_addr":"172.21.15.167:6818/1642543604","cluster_addr":"172.21.15.167:6821/1642543604","heartbeat_back_addr":"172.21.15.167:6828/1642543604","heartbeat_front_addr":"172.21.15.167:6824/1642543604","state":["exists","up"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6809","nonce":172726194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6811","nonce":172726194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6815","nonce":172726194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6813","nonce":172726194}]},"public_addr":"172.21.15.167:6809/172726194","cluster_addr":"172.21.15.167:6811/172726194","heartbeat_back_addr":"172.21.15.167:6815/172726194","heartbeat_front_addr":"172.21.15.167:6813/172726194","state":["exists","up"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6801","nonce":2195746987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6803","nonce":2195746987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6807","nonce":2195746987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6805","nonce":2195746987}]},"public_addr":"172.21.15.167:6801/2195746987","cluster_addr":"172.21.15.167:6803/2195746987","heartbeat_back_addr":"172.21.15.167:6807/2195746987","heartbeat_front_addr":"172.21.15.167:6805/2195746987","state":["exists","up"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6817","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6819","nonce":954791674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6825","nonce":954791674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6831","nonce":954791674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6827","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6829","nonce":954791674}]},"public_addr":"172.21.15.167:6819/954791674","cluster_addr":"172.21.15.167:6825/954791674","heartbeat_back_addr":"172.21.15.167:6831/954791674","heartbeat_front_addr":"172.21.15.167:6829/954791674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.509233+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.465696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.405177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.269202+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.359180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.351834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.490127+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.484842+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:33:43.449 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-24T00:33:43.449 INFO:tasks.ceph:Creating RBD pool 2024-04-24T00:33:43.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-24T00:33:43.945 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2024-04-24T00:33:43.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T00:33:44.955 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T00:33:44.983 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-24T00:33:44.983 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T00:33:44.984 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-24T00:33:44.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-24T00:33:44.984 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:45.351 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":889001083},{"type":"v1","addr":"172.21.15.167:6833","nonce":889001083}]},"active_addr":"172.21.15.167:6833/889001083","active_change":"2024-04-24T00:33:29.634373+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":1407985834}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3610312149}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":1404811638}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":188757515}]}]} 2024-04-24T00:33:45.357 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-24T00:33:45.357 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T00:33:45.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":19,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:33:44.944195+0000","last_up_change":"2024-04-24T00:33:37.880801+0000","last_in_change":"2024-04-24T00:33:27.862090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:33:39.734814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:33:43.786311+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6817","nonce":3307242121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6819","nonce":3307242121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6823","nonce":3307242121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6821","nonce":3307242121}]},"public_addr":"172.21.15.64:6817/3307242121","cluster_addr":"172.21.15.64:6819/3307242121","heartbeat_back_addr":"172.21.15.64:6823/3307242121","heartbeat_front_addr":"172.21.15.64:6821/3307242121","state":["exists","up"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6825","nonce":637720411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6827","nonce":637720411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6831","nonce":637720411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6829","nonce":637720411}]},"public_addr":"172.21.15.64:6825/637720411","cluster_addr":"172.21.15.64:6827/637720411","heartbeat_back_addr":"172.21.15.64:6831/637720411","heartbeat_front_addr":"172.21.15.64:6829/637720411","state":["exists","up"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6809","nonce":820024667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6811","nonce":820024667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6815","nonce":820024667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6813","nonce":820024667}]},"public_addr":"172.21.15.64:6809/820024667","cluster_addr":"172.21.15.64:6811/820024667","heartbeat_back_addr":"172.21.15.64:6815/820024667","heartbeat_front_addr":"172.21.15.64:6813/820024667","state":["exists","up"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6801","nonce":1974534323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6803","nonce":1974534323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6807","nonce":1974534323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6805","nonce":1974534323}]},"public_addr":"172.21.15.64:6801/1974534323","cluster_addr":"172.21.15.64:6803/1974534323","heartbeat_back_addr":"172.21.15.64:6807/1974534323","heartbeat_front_addr":"172.21.15.64:6805/1974534323","state":["exists","up"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6818","nonce":1642543604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6821","nonce":1642543604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6828","nonce":1642543604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6823","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6824","nonce":1642543604}]},"public_addr":"172.21.15.167:6818/1642543604","cluster_addr":"172.21.15.167:6821/1642543604","heartbeat_back_addr":"172.21.15.167:6828/1642543604","heartbeat_front_addr":"172.21.15.167:6824/1642543604","state":["exists","up"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6809","nonce":172726194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6811","nonce":172726194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6815","nonce":172726194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6813","nonce":172726194}]},"public_addr":"172.21.15.167:6809/172726194","cluster_addr":"172.21.15.167:6811/172726194","heartbeat_back_addr":"172.21.15.167:6815/172726194","heartbeat_front_addr":"172.21.15.167:6813/172726194","state":["exists","up"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6801","nonce":2195746987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6803","nonce":2195746987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6807","nonce":2195746987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6805","nonce":2195746987}]},"public_addr":"172.21.15.167:6801/2195746987","cluster_addr":"172.21.15.167:6803/2195746987","heartbeat_back_addr":"172.21.15.167:6807/2195746987","heartbeat_front_addr":"172.21.15.167:6805/2195746987","state":["exists","up"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6817","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6819","nonce":954791674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6825","nonce":954791674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6831","nonce":954791674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6827","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6829","nonce":954791674}]},"public_addr":"172.21.15.167:6819/954791674","cluster_addr":"172.21.15.167:6825/954791674","heartbeat_back_addr":"172.21.15.167:6831/954791674","heartbeat_front_addr":"172.21.15.167:6829/954791674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.509233+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.465696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.405177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.269202+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.359180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.351834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.490127+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.484842+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:33:45.685 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T00:33:45.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T00:33:45.994 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:45.994 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":20,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:33:45.956398+0000","last_up_change":"2024-04-24T00:33:37.880801+0000","last_in_change":"2024-04-24T00:33:27.862090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:33:39.734814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:33:43.786311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6817","nonce":3307242121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6819","nonce":3307242121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6823","nonce":3307242121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":3307242121},{"type":"v1","addr":"172.21.15.64:6821","nonce":3307242121}]},"public_addr":"172.21.15.64:6817/3307242121","cluster_addr":"172.21.15.64:6819/3307242121","heartbeat_back_addr":"172.21.15.64:6823/3307242121","heartbeat_front_addr":"172.21.15.64:6821/3307242121","state":["exists","up"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6825","nonce":637720411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6827","nonce":637720411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6831","nonce":637720411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6829","nonce":637720411}]},"public_addr":"172.21.15.64:6825/637720411","cluster_addr":"172.21.15.64:6827/637720411","heartbeat_back_addr":"172.21.15.64:6831/637720411","heartbeat_front_addr":"172.21.15.64:6829/637720411","state":["exists","up"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6809","nonce":820024667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6811","nonce":820024667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6815","nonce":820024667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6813","nonce":820024667}]},"public_addr":"172.21.15.64:6809/820024667","cluster_addr":"172.21.15.64:6811/820024667","heartbeat_back_addr":"172.21.15.64:6815/820024667","heartbeat_front_addr":"172.21.15.64:6813/820024667","state":["exists","up"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6801","nonce":1974534323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6803","nonce":1974534323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6807","nonce":1974534323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6805","nonce":1974534323}]},"public_addr":"172.21.15.64:6801/1974534323","cluster_addr":"172.21.15.64:6803/1974534323","heartbeat_back_addr":"172.21.15.64:6807/1974534323","heartbeat_front_addr":"172.21.15.64:6805/1974534323","state":["exists","up"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6818","nonce":1642543604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6821","nonce":1642543604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6828","nonce":1642543604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6823","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6824","nonce":1642543604}]},"public_addr":"172.21.15.167:6818/1642543604","cluster_addr":"172.21.15.167:6821/1642543604","heartbeat_back_addr":"172.21.15.167:6828/1642543604","heartbeat_front_addr":"172.21.15.167:6824/1642543604","state":["exists","up"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6809","nonce":172726194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6811","nonce":172726194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6815","nonce":172726194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6813","nonce":172726194}]},"public_addr":"172.21.15.167:6809/172726194","cluster_addr":"172.21.15.167:6811/172726194","heartbeat_back_addr":"172.21.15.167:6815/172726194","heartbeat_front_addr":"172.21.15.167:6813/172726194","state":["exists","up"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6801","nonce":2195746987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6803","nonce":2195746987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6807","nonce":2195746987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6805","nonce":2195746987}]},"public_addr":"172.21.15.167:6801/2195746987","cluster_addr":"172.21.15.167:6803/2195746987","heartbeat_back_addr":"172.21.15.167:6807/2195746987","heartbeat_front_addr":"172.21.15.167:6805/2195746987","state":["exists","up"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6817","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6819","nonce":954791674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6825","nonce":954791674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6831","nonce":954791674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6827","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6829","nonce":954791674}]},"public_addr":"172.21.15.167:6819/954791674","cluster_addr":"172.21.15.167:6825/954791674","heartbeat_back_addr":"172.21.15.167:6831/954791674","heartbeat_front_addr":"172.21.15.167:6829/954791674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.509233+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.465696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.405177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.269202+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.359180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.351834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.490127+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.484842+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:33:46.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T00:33:46.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T00:33:46.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T00:33:46.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T00:33:46.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T00:33:46.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T00:33:46.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-24T00:33:46.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-24T00:33:46.284 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-04-24T00:33:46.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T00:33:46.291 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:46.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T00:33:46.296 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-04-24T00:33:46.296 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T00:33:46.297 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:46.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T00:33:46.299 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-04-24T00:33:46.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T00:33:46.300 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:46.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T00:33:46.302 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-04-24T00:33:46.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T00:33:46.309 INFO:teuthology.orchestra.run.smithi064.stdout:55834574852 2024-04-24T00:33:46.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T00:33:46.817 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:46.819 INFO:teuthology.orchestra.run.smithi064.stdout:55834574852 2024-04-24T00:33:46.832 INFO:teuthology.orchestra.run.smithi064.stdout:55834574852 2024-04-24T00:33:46.833 INFO:teuthology.orchestra.run.smithi064.stdout:55834574852 2024-04-24T00:33:46.845 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:46.848 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-04-24T00:33:46.852 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-04-24T00:33:46.852 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:46.853 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:46.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-04-24T00:33:46.868 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.6 2024-04-24T00:33:46.876 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-04-24T00:33:46.877 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-04-24T00:33:46.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-04-24T00:33:46.884 INFO:teuthology.orchestra.run.smithi064.stdout:55834574851 2024-04-24T00:33:46.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2024-04-24T00:33:47.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T00:33:47.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T00:33:47.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T00:33:47.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T00:33:47.877 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T00:33:47.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T00:33:47.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T00:33:47.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T00:33:48.384 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-04-24T00:33:48.387 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:48.413 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-04-24T00:33:48.414 DEBUG:teuthology.parallel:result is None 2024-04-24T00:33:48.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-04-24T00:33:48.414 DEBUG:teuthology.parallel:result is None 2024-04-24T00:33:48.424 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:48.427 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-04-24T00:33:48.435 INFO:teuthology.orchestra.run.smithi064.stdout:55834574853 2024-04-24T00:33:48.446 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2024-04-24T00:33:48.446 DEBUG:teuthology.parallel:result is None 2024-04-24T00:33:48.450 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-04-24T00:33:48.451 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-04-24T00:33:48.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-04-24T00:33:48.457 DEBUG:teuthology.parallel:result is None 2024-04-24T00:33:48.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-04-24T00:33:48.460 DEBUG:teuthology.parallel:result is None 2024-04-24T00:33:48.472 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-04-24T00:33:48.473 DEBUG:teuthology.parallel:result is None 2024-04-24T00:33:48.479 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-04-24T00:33:48.479 DEBUG:teuthology.parallel:result is None 2024-04-24T00:33:48.482 INFO:teuthology.orchestra.run.smithi064.stdout:55834574852 2024-04-24T00:33:48.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2024-04-24T00:33:48.499 DEBUG:teuthology.parallel:result is None 2024-04-24T00:33:48.499 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T00:33:48.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:33:48.817 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:48.818 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:33:48.834 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-24T00:33:47.647530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217768,"kb_used_data":1592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756952,"statfs":{"total":773094113280,"available":772871118848,"internally_reserved":0,"allocated":1630208,"data_stored":1665056,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.703321"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-24T00:33:44.963714+0000","last_change":"2024-04-24T00:33:44.961547+0000","last_active":"2024-04-24T00:33:44.963714+0000","last_peered":"2024-04-24T00:33:44.963714+0000","last_clean":"2024-04-24T00:33:44.963714+0000","last_became_active":"2024-04-24T00:33:44.960683+0000","last_became_peered":"2024-04-24T00:33:44.960683+0000","last_unstale":"2024-04-24T00:33:44.963714+0000","last_undegraded":"2024-04-24T00:33:44.963714+0000","last_fullsized":"2024-04-24T00:33:44.963714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:56:47.341358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.963635+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:33:45.963635+0000","last_peered":"2024-04-24T00:33:45.963635+0000","last_clean":"2024-04-24T00:33:45.963635+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:33:45.963635+0000","last_undegraded":"2024-04-24T00:33:45.963635+0000","last_fullsized":"2024-04-24T00:33:45.963635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.649789+0000","last_change":"2024-04-24T00:33:45.648584+0000","last_active":"2024-04-24T00:33:45.649789+0000","last_peered":"2024-04-24T00:33:45.649789+0000","last_clean":"2024-04-24T00:33:45.649789+0000","last_became_active":"2024-04-24T00:33:45.647889+0000","last_became_peered":"2024-04-24T00:33:45.647889+0000","last_unstale":"2024-04-24T00:33:45.649789+0000","last_undegraded":"2024-04-24T00:33:45.649789+0000","last_fullsized":"2024-04-24T00:33:45.649789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:58.109051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.963927+0000","last_change":"2024-04-24T00:33:45.650424+0000","last_active":"2024-04-24T00:33:45.963927+0000","last_peered":"2024-04-24T00:33:45.963927+0000","last_clean":"2024-04-24T00:33:45.963927+0000","last_became_active":"2024-04-24T00:33:45.649310+0000","last_became_peered":"2024-04-24T00:33:45.649310+0000","last_unstale":"2024-04-24T00:33:45.963927+0000","last_undegraded":"2024-04-24T00:33:45.963927+0000","last_fullsized":"2024-04-24T00:33:45.963927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:40:40.748700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.961405+0000","last_change":"2024-04-24T00:33:44.963349+0000","last_active":"2024-04-24T00:33:45.961405+0000","last_peered":"2024-04-24T00:33:45.961405+0000","last_clean":"2024-04-24T00:33:45.961405+0000","last_became_active":"2024-04-24T00:33:44.962580+0000","last_became_peered":"2024-04-24T00:33:44.962580+0000","last_unstale":"2024-04-24T00:33:45.961405+0000","last_undegraded":"2024-04-24T00:33:45.961405+0000","last_fullsized":"2024-04-24T00:33:45.961405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.965200+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:33:45.965200+0000","last_peered":"2024-04-24T00:33:45.965200+0000","last_clean":"2024-04-24T00:33:45.965200+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:33:45.965200+0000","last_undegraded":"2024-04-24T00:33:45.965200+0000","last_fullsized":"2024-04-24T00:33:45.965200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-24T00:33:44.965420+0000","last_change":"2024-04-24T00:33:44.961018+0000","last_active":"2024-04-24T00:33:44.965420+0000","last_peered":"2024-04-24T00:33:44.965420+0000","last_clean":"2024-04-24T00:33:44.965420+0000","last_became_active":"2024-04-24T00:33:44.959946+0000","last_became_peered":"2024-04-24T00:33:44.959946+0000","last_unstale":"2024-04-24T00:33:44.965420+0000","last_undegraded":"2024-04-24T00:33:44.965420+0000","last_fullsized":"2024-04-24T00:33:44.965420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:44.027330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.961382+0000","last_change":"2024-04-24T00:33:44.962772+0000","last_active":"2024-04-24T00:33:45.961382+0000","last_peered":"2024-04-24T00:33:45.961382+0000","last_clean":"2024-04-24T00:33:45.961382+0000","last_became_active":"2024-04-24T00:33:44.961868+0000","last_became_peered":"2024-04-24T00:33:44.961868+0000","last_unstale":"2024-04-24T00:33:45.961382+0000","last_undegraded":"2024-04-24T00:33:45.961382+0000","last_fullsized":"2024-04-24T00:33:45.961382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:01.330023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-24T00:33:44.954872+0000","last_change":"2024-04-24T00:33:41.026859+0000","last_active":"2024-04-24T00:33:44.954872+0000","last_peered":"2024-04-24T00:33:44.954872+0000","last_clean":"2024-04-24T00:33:44.954872+0000","last_became_active":"2024-04-24T00:33:41.024668+0000","last_became_peered":"2024-04-24T00:33:41.024668+0000","last_unstale":"2024-04-24T00:33:44.954872+0000","last_undegraded":"2024-04-24T00:33:44.954872+0000","last_fullsized":"2024-04-24T00:33:44.954872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:25.500752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":652355,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":56199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":646567,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:33:48.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:33:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:33:49.144 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-24T00:33:47.647530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217768,"kb_used_data":1592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756952,"statfs":{"total":773094113280,"available":772871118848,"internally_reserved":0,"allocated":1630208,"data_stored":1665056,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.703321"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-24T00:33:44.963714+0000","last_change":"2024-04-24T00:33:44.961547+0000","last_active":"2024-04-24T00:33:44.963714+0000","last_peered":"2024-04-24T00:33:44.963714+0000","last_clean":"2024-04-24T00:33:44.963714+0000","last_became_active":"2024-04-24T00:33:44.960683+0000","last_became_peered":"2024-04-24T00:33:44.960683+0000","last_unstale":"2024-04-24T00:33:44.963714+0000","last_undegraded":"2024-04-24T00:33:44.963714+0000","last_fullsized":"2024-04-24T00:33:44.963714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:56:47.341358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.963635+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:33:45.963635+0000","last_peered":"2024-04-24T00:33:45.963635+0000","last_clean":"2024-04-24T00:33:45.963635+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:33:45.963635+0000","last_undegraded":"2024-04-24T00:33:45.963635+0000","last_fullsized":"2024-04-24T00:33:45.963635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.649789+0000","last_change":"2024-04-24T00:33:45.648584+0000","last_active":"2024-04-24T00:33:45.649789+0000","last_peered":"2024-04-24T00:33:45.649789+0000","last_clean":"2024-04-24T00:33:45.649789+0000","last_became_active":"2024-04-24T00:33:45.647889+0000","last_became_peered":"2024-04-24T00:33:45.647889+0000","last_unstale":"2024-04-24T00:33:45.649789+0000","last_undegraded":"2024-04-24T00:33:45.649789+0000","last_fullsized":"2024-04-24T00:33:45.649789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:58.109051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.963927+0000","last_change":"2024-04-24T00:33:45.650424+0000","last_active":"2024-04-24T00:33:45.963927+0000","last_peered":"2024-04-24T00:33:45.963927+0000","last_clean":"2024-04-24T00:33:45.963927+0000","last_became_active":"2024-04-24T00:33:45.649310+0000","last_became_peered":"2024-04-24T00:33:45.649310+0000","last_unstale":"2024-04-24T00:33:45.963927+0000","last_undegraded":"2024-04-24T00:33:45.963927+0000","last_fullsized":"2024-04-24T00:33:45.963927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:40:40.748700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.961405+0000","last_change":"2024-04-24T00:33:44.963349+0000","last_active":"2024-04-24T00:33:45.961405+0000","last_peered":"2024-04-24T00:33:45.961405+0000","last_clean":"2024-04-24T00:33:45.961405+0000","last_became_active":"2024-04-24T00:33:44.962580+0000","last_became_peered":"2024-04-24T00:33:44.962580+0000","last_unstale":"2024-04-24T00:33:45.961405+0000","last_undegraded":"2024-04-24T00:33:45.961405+0000","last_fullsized":"2024-04-24T00:33:45.961405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.965200+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:33:45.965200+0000","last_peered":"2024-04-24T00:33:45.965200+0000","last_clean":"2024-04-24T00:33:45.965200+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:33:45.965200+0000","last_undegraded":"2024-04-24T00:33:45.965200+0000","last_fullsized":"2024-04-24T00:33:45.965200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-24T00:33:44.965420+0000","last_change":"2024-04-24T00:33:44.961018+0000","last_active":"2024-04-24T00:33:44.965420+0000","last_peered":"2024-04-24T00:33:44.965420+0000","last_clean":"2024-04-24T00:33:44.965420+0000","last_became_active":"2024-04-24T00:33:44.959946+0000","last_became_peered":"2024-04-24T00:33:44.959946+0000","last_unstale":"2024-04-24T00:33:44.965420+0000","last_undegraded":"2024-04-24T00:33:44.965420+0000","last_fullsized":"2024-04-24T00:33:44.965420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:44.027330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.961382+0000","last_change":"2024-04-24T00:33:44.962772+0000","last_active":"2024-04-24T00:33:45.961382+0000","last_peered":"2024-04-24T00:33:45.961382+0000","last_clean":"2024-04-24T00:33:45.961382+0000","last_became_active":"2024-04-24T00:33:44.961868+0000","last_became_peered":"2024-04-24T00:33:44.961868+0000","last_unstale":"2024-04-24T00:33:45.961382+0000","last_undegraded":"2024-04-24T00:33:45.961382+0000","last_fullsized":"2024-04-24T00:33:45.961382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:01.330023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-24T00:33:44.954872+0000","last_change":"2024-04-24T00:33:41.026859+0000","last_active":"2024-04-24T00:33:44.954872+0000","last_peered":"2024-04-24T00:33:44.954872+0000","last_clean":"2024-04-24T00:33:44.954872+0000","last_became_active":"2024-04-24T00:33:41.024668+0000","last_became_peered":"2024-04-24T00:33:41.024668+0000","last_unstale":"2024-04-24T00:33:44.954872+0000","last_undegraded":"2024-04-24T00:33:44.954872+0000","last_fullsized":"2024-04-24T00:33:44.954872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:25.500752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":652355,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":56199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":61987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":646567,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:33:49.145 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T00:33:49.145 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T00:33:49.146 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-24T00:33:49.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T00:33:49.468 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:49.469 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T00:33:49.486 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-24T00:33:49.486 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-24T00:33:49.500 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-24T00:33:49.500 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}} 2024-04-24T00:33:49.500 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-24T00:33:49.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-24T00:33:49.814 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-24T00:33:49.814 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T00:33:49.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:33:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:50.102 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:33:50.118 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-24T00:33:49.648290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217900,"kb_used_data":1712,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756820,"statfs":{"total":773094113280,"available":772870983680,"internally_reserved":0,"allocated":1753088,"data_stored":1722616,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.704081"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:47.424050+0000","last_change":"2024-04-24T00:33:44.961547+0000","last_active":"2024-04-24T00:33:47.424050+0000","last_peered":"2024-04-24T00:33:47.424050+0000","last_clean":"2024-04-24T00:33:47.424050+0000","last_became_active":"2024-04-24T00:33:44.960683+0000","last_became_peered":"2024-04-24T00:33:44.960683+0000","last_unstale":"2024-04-24T00:33:47.424050+0000","last_undegraded":"2024-04-24T00:33:47.424050+0000","last_fullsized":"2024-04-24T00:33:47.424050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.963635+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:33:45.963635+0000","last_peered":"2024-04-24T00:33:45.963635+0000","last_clean":"2024-04-24T00:33:45.963635+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:33:45.963635+0000","last_undegraded":"2024-04-24T00:33:45.963635+0000","last_fullsized":"2024-04-24T00:33:45.963635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:47.423761+0000","last_change":"2024-04-24T00:33:45.648584+0000","last_active":"2024-04-24T00:33:47.423761+0000","last_peered":"2024-04-24T00:33:47.423761+0000","last_clean":"2024-04-24T00:33:47.423761+0000","last_became_active":"2024-04-24T00:33:45.647889+0000","last_became_peered":"2024-04-24T00:33:45.647889+0000","last_unstale":"2024-04-24T00:33:47.423761+0000","last_undegraded":"2024-04-24T00:33:47.423761+0000","last_fullsized":"2024-04-24T00:33:47.423761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:52.085162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.963927+0000","last_change":"2024-04-24T00:33:45.650424+0000","last_active":"2024-04-24T00:33:45.963927+0000","last_peered":"2024-04-24T00:33:45.963927+0000","last_clean":"2024-04-24T00:33:45.963927+0000","last_became_active":"2024-04-24T00:33:45.649310+0000","last_became_peered":"2024-04-24T00:33:45.649310+0000","last_unstale":"2024-04-24T00:33:45.963927+0000","last_undegraded":"2024-04-24T00:33:45.963927+0000","last_fullsized":"2024-04-24T00:33:45.963927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:40:40.748700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.961405+0000","last_change":"2024-04-24T00:33:44.963349+0000","last_active":"2024-04-24T00:33:45.961405+0000","last_peered":"2024-04-24T00:33:45.961405+0000","last_clean":"2024-04-24T00:33:45.961405+0000","last_became_active":"2024-04-24T00:33:44.962580+0000","last_became_peered":"2024-04-24T00:33:44.962580+0000","last_unstale":"2024-04-24T00:33:45.961405+0000","last_undegraded":"2024-04-24T00:33:45.961405+0000","last_fullsized":"2024-04-24T00:33:45.961405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.965200+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:33:45.965200+0000","last_peered":"2024-04-24T00:33:45.965200+0000","last_clean":"2024-04-24T00:33:45.965200+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:33:45.965200+0000","last_undegraded":"2024-04-24T00:33:45.965200+0000","last_fullsized":"2024-04-24T00:33:45.965200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:47.423674+0000","last_change":"2024-04-24T00:33:44.961018+0000","last_active":"2024-04-24T00:33:47.423674+0000","last_peered":"2024-04-24T00:33:47.423674+0000","last_clean":"2024-04-24T00:33:47.423674+0000","last_became_active":"2024-04-24T00:33:44.959946+0000","last_became_peered":"2024-04-24T00:33:44.959946+0000","last_unstale":"2024-04-24T00:33:47.423674+0000","last_undegraded":"2024-04-24T00:33:47.423674+0000","last_fullsized":"2024-04-24T00:33:47.423674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:12.966651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.961382+0000","last_change":"2024-04-24T00:33:44.962772+0000","last_active":"2024-04-24T00:33:45.961382+0000","last_peered":"2024-04-24T00:33:45.961382+0000","last_clean":"2024-04-24T00:33:45.961382+0000","last_became_active":"2024-04-24T00:33:44.961868+0000","last_became_peered":"2024-04-24T00:33:44.961868+0000","last_unstale":"2024-04-24T00:33:45.961382+0000","last_undegraded":"2024-04-24T00:33:45.961382+0000","last_fullsized":"2024-04-24T00:33:45.961382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:01.330023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:47.423589+0000","last_change":"2024-04-24T00:33:41.026859+0000","last_active":"2024-04-24T00:33:47.423589+0000","last_peered":"2024-04-24T00:33:47.423589+0000","last_clean":"2024-04-24T00:33:47.423589+0000","last_became_active":"2024-04-24T00:33:41.024668+0000","last_became_peered":"2024-04-24T00:33:41.024668+0000","last_unstale":"2024-04-24T00:33:47.423589+0000","last_undegraded":"2024-04-24T00:33:47.423589+0000","last_fullsized":"2024-04-24T00:33:47.423589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:25.500752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":658103,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":658103,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:33:50.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:33:50.391 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:33:50.392 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:33:50.407 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-24T00:33:49.648290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217900,"kb_used_data":1712,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756820,"statfs":{"total":773094113280,"available":772870983680,"internally_reserved":0,"allocated":1753088,"data_stored":1722616,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.704081"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:47.424050+0000","last_change":"2024-04-24T00:33:44.961547+0000","last_active":"2024-04-24T00:33:47.424050+0000","last_peered":"2024-04-24T00:33:47.424050+0000","last_clean":"2024-04-24T00:33:47.424050+0000","last_became_active":"2024-04-24T00:33:44.960683+0000","last_became_peered":"2024-04-24T00:33:44.960683+0000","last_unstale":"2024-04-24T00:33:47.424050+0000","last_undegraded":"2024-04-24T00:33:47.424050+0000","last_fullsized":"2024-04-24T00:33:47.424050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.963635+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:33:45.963635+0000","last_peered":"2024-04-24T00:33:45.963635+0000","last_clean":"2024-04-24T00:33:45.963635+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:33:45.963635+0000","last_undegraded":"2024-04-24T00:33:45.963635+0000","last_fullsized":"2024-04-24T00:33:45.963635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:47.423761+0000","last_change":"2024-04-24T00:33:45.648584+0000","last_active":"2024-04-24T00:33:47.423761+0000","last_peered":"2024-04-24T00:33:47.423761+0000","last_clean":"2024-04-24T00:33:47.423761+0000","last_became_active":"2024-04-24T00:33:45.647889+0000","last_became_peered":"2024-04-24T00:33:45.647889+0000","last_unstale":"2024-04-24T00:33:47.423761+0000","last_undegraded":"2024-04-24T00:33:47.423761+0000","last_fullsized":"2024-04-24T00:33:47.423761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:52.085162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.963927+0000","last_change":"2024-04-24T00:33:45.650424+0000","last_active":"2024-04-24T00:33:45.963927+0000","last_peered":"2024-04-24T00:33:45.963927+0000","last_clean":"2024-04-24T00:33:45.963927+0000","last_became_active":"2024-04-24T00:33:45.649310+0000","last_became_peered":"2024-04-24T00:33:45.649310+0000","last_unstale":"2024-04-24T00:33:45.963927+0000","last_undegraded":"2024-04-24T00:33:45.963927+0000","last_fullsized":"2024-04-24T00:33:45.963927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:40:40.748700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.961405+0000","last_change":"2024-04-24T00:33:44.963349+0000","last_active":"2024-04-24T00:33:45.961405+0000","last_peered":"2024-04-24T00:33:45.961405+0000","last_clean":"2024-04-24T00:33:45.961405+0000","last_became_active":"2024-04-24T00:33:44.962580+0000","last_became_peered":"2024-04-24T00:33:44.962580+0000","last_unstale":"2024-04-24T00:33:45.961405+0000","last_undegraded":"2024-04-24T00:33:45.961405+0000","last_fullsized":"2024-04-24T00:33:45.961405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.965200+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:33:45.965200+0000","last_peered":"2024-04-24T00:33:45.965200+0000","last_clean":"2024-04-24T00:33:45.965200+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:33:45.965200+0000","last_undegraded":"2024-04-24T00:33:45.965200+0000","last_fullsized":"2024-04-24T00:33:45.965200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:47.423674+0000","last_change":"2024-04-24T00:33:44.961018+0000","last_active":"2024-04-24T00:33:47.423674+0000","last_peered":"2024-04-24T00:33:47.423674+0000","last_clean":"2024-04-24T00:33:47.423674+0000","last_became_active":"2024-04-24T00:33:44.959946+0000","last_became_peered":"2024-04-24T00:33:44.959946+0000","last_unstale":"2024-04-24T00:33:47.423674+0000","last_undegraded":"2024-04-24T00:33:47.423674+0000","last_fullsized":"2024-04-24T00:33:47.423674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:12.966651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:45.961382+0000","last_change":"2024-04-24T00:33:44.962772+0000","last_active":"2024-04-24T00:33:45.961382+0000","last_peered":"2024-04-24T00:33:45.961382+0000","last_clean":"2024-04-24T00:33:45.961382+0000","last_became_active":"2024-04-24T00:33:44.961868+0000","last_became_peered":"2024-04-24T00:33:44.961868+0000","last_unstale":"2024-04-24T00:33:45.961382+0000","last_undegraded":"2024-04-24T00:33:45.961382+0000","last_fullsized":"2024-04-24T00:33:45.961382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:17:01.330023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T00:33:47.423589+0000","last_change":"2024-04-24T00:33:41.026859+0000","last_active":"2024-04-24T00:33:47.423589+0000","last_peered":"2024-04-24T00:33:47.423589+0000","last_clean":"2024-04-24T00:33:47.423589+0000","last_became_active":"2024-04-24T00:33:41.024668+0000","last_became_peered":"2024-04-24T00:33:41.024668+0000","last_unstale":"2024-04-24T00:33:47.423589+0000","last_undegraded":"2024-04-24T00:33:47.423589+0000","last_fullsized":"2024-04-24T00:33:47.423589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:25.500752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":658103,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":67735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":658103,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:33:50.408 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T00:33:50.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-24T00:33:50.704 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 20 2024-04-24T00:33:50.704 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 6a70d87d-4453-47be-af2a-5590f9851254 2024-04-24T00:33:50.704 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-04-24T00:33:24.857711+0000 2024-04-24T00:33:50.704 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2024-04-24T00:33:45.956398+0000 2024-04-24T00:33:50.704 INFO:teuthology.orchestra.run.smithi064.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-24T00:33:50.704 INFO:teuthology.orchestra.run.smithi064.stdout:crush_version 4 2024-04-24T00:33:50.704 INFO:teuthology.orchestra.run.smithi064.stdout:full_ratio 0.95 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:backfillfull_ratio 0.9 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:nearfull_ratio 0.85 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:require_min_compat_client luminous 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:min_compat_client jewel 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:require_osd_release squid 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:stretch_mode_enabled false 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 application rbd read_balance_score 1.99 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:max_osd 8 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6816/3307242121,v1:172.21.15.64:6817/3307242121] [v2:172.21.15.64:6818/3307242121,v1:172.21.15.64:6819/3307242121] exists,up 1c80b3f4-99ee-4725-90a8-920116161bf5 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6824/637720411,v1:172.21.15.64:6825/637720411] [v2:172.21.15.64:6826/637720411,v1:172.21.15.64:6827/637720411] exists,up 20e1331a-f8a2-48fd-b2b8-b7a92f21d78e 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/820024667,v1:172.21.15.64:6809/820024667] [v2:172.21.15.64:6810/820024667,v1:172.21.15.64:6811/820024667] exists,up a51557a0-9634-4a20-82b0-b54cdd36b166 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/1974534323,v1:172.21.15.64:6801/1974534323] [v2:172.21.15.64:6802/1974534323,v1:172.21.15.64:6803/1974534323] exists,up aafffa79-f532-4441-b1d6-f6441c5eebf8 2024-04-24T00:33:50.705 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.167:6816/1642543604,v1:172.21.15.167:6818/1642543604] [v2:172.21.15.167:6820/1642543604,v1:172.21.15.167:6821/1642543604] exists,up 78f05556-598e-4be2-b682-e3f58863eb82 2024-04-24T00:33:50.706 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.167:6808/172726194,v1:172.21.15.167:6809/172726194] [v2:172.21.15.167:6810/172726194,v1:172.21.15.167:6811/172726194] exists,up 65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5 2024-04-24T00:33:50.706 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.167:6800/2195746987,v1:172.21.15.167:6801/2195746987] [v2:172.21.15.167:6802/2195746987,v1:172.21.15.167:6803/2195746987] exists,up 6dad12bf-94de-482a-a402-49e67d5e5dc7 2024-04-24T00:33:50.706 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.167:6817/954791674,v1:172.21.15.167:6819/954791674] [v2:172.21.15.167:6822/954791674,v1:172.21.15.167:6825/954791674] exists,up e6734cfe-3c24-4ddd-8ef0-5ed450efe772 2024-04-24T00:33:50.723 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6816/3307242121,v1:172.21.15.64:6817/3307242121] [v2:172.21.15.64:6818/3307242121,v1:172.21.15.64:6819/3307242121] exists,up 1c80b3f4-99ee-4725-90a8-920116161bf5', 'osd.1 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6824/637720411,v1:172.21.15.64:6825/637720411] [v2:172.21.15.64:6826/637720411,v1:172.21.15.64:6827/637720411] exists,up 20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'osd.2 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/820024667,v1:172.21.15.64:6809/820024667] [v2:172.21.15.64:6810/820024667,v1:172.21.15.64:6811/820024667] exists,up a51557a0-9634-4a20-82b0-b54cdd36b166', 'osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/1974534323,v1:172.21.15.64:6801/1974534323] [v2:172.21.15.64:6802/1974534323,v1:172.21.15.64:6803/1974534323] exists,up aafffa79-f532-4441-b1d6-f6441c5eebf8', 'osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.167:6816/1642543604,v1:172.21.15.167:6818/1642543604] [v2:172.21.15.167:6820/1642543604,v1:172.21.15.167:6821/1642543604] exists,up 78f05556-598e-4be2-b682-e3f58863eb82', 'osd.5 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.167:6808/172726194,v1:172.21.15.167:6809/172726194] [v2:172.21.15.167:6810/172726194,v1:172.21.15.167:6811/172726194] exists,up 65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'osd.6 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.167:6800/2195746987,v1:172.21.15.167:6801/2195746987] [v2:172.21.15.167:6802/2195746987,v1:172.21.15.167:6803/2195746987] exists,up 6dad12bf-94de-482a-a402-49e67d5e5dc7', 'osd.7 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.167:6817/954791674,v1:172.21.15.167:6819/954791674] [v2:172.21.15.167:6822/954791674,v1:172.21.15.167:6825/954791674] exists,up e6734cfe-3c24-4ddd-8ef0-5ed450efe772'] 2024-04-24T00:33:50.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-24T00:33:50.844 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:50.844 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mon.a", 2024-04-24T00:33:50.844 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster": "ceph", 2024-04-24T00:33:50.844 INFO:teuthology.orchestra.run.smithi064.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-24T00:33:50.844 INFO:teuthology.orchestra.run.smithi064.stdout: "admin_socket_mode": "", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_client_required": "cephx, none", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_cluster_required": "cephx", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_debug": "true", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_service_required": "cephx", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_supported": "cephx", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio": "true", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_poll_ms": "250", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_reap_max": "16", 2024-04-24T00:33:50.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_async_discard": "false", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_block_size": "4096", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio": "false", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_inflight_ios": "false", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_enable_discard": "false", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_flock_retry": "3", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_inject_crash": "0", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring": "false", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring_hipri": "false", 2024-04-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_type": "", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_alloc_size": "1048576", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_allocator": "hybrid", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_buffered_io": "true", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_for_zeros": "false", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_compact_log_sync": "false", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_max_log_runway": "4194304", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_max_prefetch": "1048576", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_min_flush_size": "524288", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_min_log_runway": "1048576", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_replay_recovery": "false", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_sync_write": "false", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_allocation_from_file": "true", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_allocator": "hybrid", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_create": "true", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_create": "false", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_path": "", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_size": "0", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_path": "", 2024-04-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_preallocate_file": "false", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_size": "107374182400", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_create": "false", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_path": "", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_size": "100663296", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs": "true", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_autotune": "true", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size": "0", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_type": "2q", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_clone_cow": "true", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-24T00:33:50.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_mode": "none", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_csum_type": "crc32c", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_freelist": "false", 2024-04-24T00:33:50.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_small_allocations": "0", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_default_buffered_read": "true", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_default_buffered_write": "false", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-24T00:33:50.854 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fail_eio": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mount": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_umount": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-24T00:33:50.855 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_ignore_data_csum": "false", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_op_age": "5.000000", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_alloc_size": "0", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size": "0", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-24T00:33:50.856 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_deferred_txc": "32", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size": "0", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_nid_prealloc": "1024", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_retry_disk_reads": "3", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_cf": "true", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-24T00:33:50.857 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_mem": "512", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_tracing": "false", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-24T00:33:50.858 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_zero_block_detection": "false", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-24T00:33:50.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_cluster_require_signatures": "false", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_cluster_require_version": "2", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_require_signatures": "false", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_require_version": "2", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_service_require_signatures": "false", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_service_require_version": "2", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_sign_messages": "true", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "chdir": "", 2024-04-24T00:33:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: "client_acl_type": "", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_asio_thread_count": "2", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_cache_mid": "0.750000", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_cache_size": "16384", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_caps_release_delay": "5", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_check_pool_perm": "true", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_force_sync_read": "false", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_getattr_caps": "false", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_inject_tick_delay": "0", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_die_on_failed_remount": "false", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_dirsize_rbytes": "true", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_force_lazyio": "false", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_fs": "", 2024-04-24T00:33:50.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_inject_release_failure": "false", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_max_inline_size": "4096", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mds_namespace": "", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_metadata": "", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_gid": "-1", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_timeout": "300", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_uid": "-1", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mountpoint": "/", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_notify_timeout": "10", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc": "true", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_dirty": "104857600", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-24T00:33:50.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_objects": "1000", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_size": "209715200", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_target_dirty": "8388608", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_permissions": "true", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_quota": "true", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_quota_df": "true", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_max_bytes": "0", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_max_periods": "4", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_min": "131072", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_reconnect_stale": "false", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_shutdown_timeout": "30", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_snapdir": ".snap", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_tick_interval": "1", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_trace": "", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_try_dentry_invalidate": "false", 2024-04-24T00:33:50.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_use_faked_inos": "false", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_use_random_mds": "false", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog": "false", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog_port": "12201", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_monitors": "default=true", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog": "false", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog_level": "info", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "-", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_network": "", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_network_interface": "", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_isal": "false", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_level": "5", 2024-04-24T00:33:50.864 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_winsize": "-15", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zstd_level": "1", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_alien_op_num_threads": "6", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_seastar_cpu_cores": "", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location_hook": "", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location_hook_timeout": "10", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "daemonize": "false", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "daos_pool": "tank", 2024-04-24T00:33:50.865 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_alienstore": "0/5", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_allow_any_pool_priority": "false", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asok": "1/5", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asok_assert_abort": "false", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asserts_on_shutdown": "true", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_auth": "20/20", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bdev": "1/3", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bluefs": "1/5", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bluestore": "1/5", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_buffer": "0/1", 2024-04-24T00:33:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ceph_exporter": "1/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cephfs_mirror": "0/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cephsqlite": "0/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_client": "0/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_compressor": "1/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_context": "0/1", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crimson_interrupt": "0/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crush": "1/1", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crypto": "1/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cyanstore": "0/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_deliberately_leak_memory": "false", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_disable_randomized_ping": "false", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_dpdk": "1/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_eventtrace": "1/5", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_filer": "0/1", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_filestore": "1/3", 2024-04-24T00:33:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_finisher": "1/1", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_fuse": "1/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_heartbeat_testing_span": "0", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_heartbeatmap": "1/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_javaclient": "1/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_journal": "1/3", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_journaler": "0/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_kstore": "1/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_lockdep": "0/1", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mclock": "1/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds": "1/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_balancer": "1/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_locker": "1/5", 2024-04-24T00:33:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_log": "1/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_log_expire": "1/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_migrator": "1/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_quiesce": "3/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_memstore": "1/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mgr": "2/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mgrc": "1/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mon": "20/20", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_monc": "0/10", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ms": "1/1", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_none": "0/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objclass": "0/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objectcacher": "0/5", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objecter": "0/1", 2024-04-24T00:33:50.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_optracker": "0/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_osd": "1/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_paxos": "20/20", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_perfcounter": "1/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_prioritycache": "1/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rados": "0/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd": "0/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_mirror": "0/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_pwl": "0/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_replay": "0/5", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_refs": "0/0", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_reserver": "1/1", 2024-04-24T00:33:50.870 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw": "1/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_access": "1/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_datacache": "1/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_dbstore": "1/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_flight": "1/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_sync": "1/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rocksdb": "4/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore": "0/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_backref": "0/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_cache": "0/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_cleaner": "0/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_device": "0/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_epm": "0/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-24T00:33:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_journal": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_lba": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_odata": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_omap": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_onode": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_t": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_tm": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_striper": "0/1", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_test": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_throttle": "1/1", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_timer": "0/1", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_tp": "0/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_trace": "1/5", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-24T00:33:50.872 INFO:teuthology.orchestra.run.smithi064.stdout: "device_failure_prediction_mode": "none", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_graylog": "false", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_journald": "false", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_stderr": "true", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_syslog": "false", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event_tracing": "false", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_addr": "0.0.0.0", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_http_port": "9926", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_prio_limit": "5", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_sort_metrics": "true", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_stats_period": "5", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "fake_statfs_for_testing": "0", 2024-04-24T00:33:50.873 INFO:teuthology.orchestra.run.smithi064.stdout: "fatal_signal_handlers": "true", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filer_max_purge_ops": "10", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filer_max_truncate_ops": "128", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_apply_finisher_threads": "1", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_blackhole": "false", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_btrfs_clone_range": "true", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_btrfs_snap": "true", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_caller_concurrency": "10", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_collect_device_partition_information": "true", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_commit_timeout": "600.000000", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_inject_read_err": "false", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_omap_check": "false", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-24T00:33:50.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_verify_split": "false", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_dump_file": "", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fadvise": "true", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fail_eio": "true", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fd_cache_shards": "16", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fd_cache_size": "128", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fiemap": "false", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fiemap_threshold": "4096", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_inject_stall": "0", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_parallel": "false", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_trailing": "false", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_writeahead": "false", 2024-04-24T00:33:50.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_kill_at": "0", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs": "0", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size": "0", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-24T00:33:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_merge_threshold": "-10", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_odsync_write": "false", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_backend": "rocksdb", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_backend_path": "", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_thread_timeout": "60", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_threads": "2", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_punch_hole": "false", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-24T00:33:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_ops": "50", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_seek_data_hole": "false", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_sloppy_crc": "false", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_splice": "false", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_split_multiple": "2", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_split_rand_factor": "20", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_update_to": "1000", 2024-04-24T00:33:50.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_enable": "true", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-24T00:33:50.879 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_xfs_extsize": "false", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_zfs_snap": "false", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fio_dir": "/tmp/fio", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_allow_other": "true", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_atomic_o_trunc": "true", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_big_writes": "true", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_debug": "false", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_default_permissions": "false", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_disable_pagecache": "false", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_max_write": "0", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_multithreaded": "true", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_require_active_mds": "true", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_set_user_groups": "true", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_move": "true", 2024-04-24T00:33:50.880 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_read": "true", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_write": "true", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_use_invalidate_cb": "true", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "gss_target_name": "ceph", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_file": "", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_inject_failure": "0", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_interval": "5", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "host": "smithi064", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-24T00:33:50.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "inject_early_sigterm": "false", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "jaeger_agent_port": "6799", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "jaeger_tracing_enable": "false", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_aio": "true", 2024-04-24T00:33:50.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_align_min_size": "65536", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_block_align": "true", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_block_size": "4096", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_dio": "true", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_discard": "false", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_force_aio": "false", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_ignore_corruption": "false", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_max_write_bytes": "10485760", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_max_write_entries": "100", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_replay_from": "0", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-24T00:33:50.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_write_header_frequency": "0", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_zero_on_create": "false", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_prefetch_periods": "10", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_prezero_periods": "5", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_write_head_interval": "15", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "key": "", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "keyfile": "", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_backend": "rocksdb", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_default_stripe_size": "65536", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_fsck_on_mount": "false", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_max_bytes": "67108864", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_max_ops": "512", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_nid_prealloc": "1024", 2024-04-24T00:33:50.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_onode_map_size": "1024", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_sync_submit_transaction": "false", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_sync_transaction": "false", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "librados_thread_count": "2", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "lockdep": "false", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "lockdep_force_backtrace": "false", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_coarse_timestamps": "true", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_flush_on_exit": "false", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_graylog_host": "127.0.0.1", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_graylog_port": "12201", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_max_new": "1000", 2024-04-24T00:33:50.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_max_recent": "10000", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_stderr_prefix": "", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_stop_at_utilization": "0.970000", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": "true", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_graylog": "false", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_journald": "false", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_stderr": "false", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_syslog": "false", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "max_rotating_auth_attempts": "10", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_action_on_write_error": "1", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_alternate_name_max": "8192", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_asio_thread_count": "2", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_export_pin": "true", 2024-04-24T00:33:50.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_dirs": "true", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_interval": "5", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_interval": "10", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_max": "-1", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_max_until": "-1", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_merge_size": "50", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_midchunk": "0.300000", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_min_start": "0.200000", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_minchunk": "0.001000", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_mode": "0", 2024-04-24T00:33:50.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_need_max": "1.200000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_need_min": "0.800000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_overload_epochs": "2", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_bits": "3", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_size": "10000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_target_decay": "10.000000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_grace": "15.000000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_interval": "4.000000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_mid": "0.700000", 2024-04-24T00:33:50.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_delay": "0", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_release_free_interval": "10", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_reservation": "0.050000", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_interval": "1", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_threshold": "262144", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_prealloc_inos": "1000", 2024-04-24T00:33:50.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_connect_bootstrapping": "false", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_damage_table_max_entries": "10000", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_auth_pins": "false", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_frag": "false", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_scatterstat": "false", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_subtrees": "false", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_decay_halflife": "5.000000", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_default_dir_hash": "2", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_defer_session_stale": "true", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_deny_all_reconnect": "false", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_keys_per_op": "16384", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_max_commit_size": "10", 2024-04-24T00:33:50.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_max_entries": "0", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_prefetch": "true", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_on_map": "false", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_early_reply": "true", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_enable_op_tracker": "true", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_enforce_unique_name": "true", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_random": "true", 2024-04-24T00:33:50.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_health_summarize_threshold": "10", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_health_dummy": "false", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_migrator_session_race": "false", 2024-04-24T00:33:50.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_join_fs": "", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_journal_format": "1", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_create_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_export_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_import_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_expire_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_replay_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_link_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_mdstable_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_openc_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_rename_at": "0", 2024-04-24T00:33:50.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_shutdown_at": "0", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_event_large_threshold": "524288", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_events_per_segment": "1024", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_max_events": "-1", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_max_segments": "128", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_pause": "false", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_segment_size": "0", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_threshold": "128", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-24T00:33:50.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_warn_factor": "2.000000", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_caps_per_client": "1048576", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_completed_flushes": "100000", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_completed_requests": "100000", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_export_size": "20971520", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_file_recover": "32", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_files": "64", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_ops": "8192", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_snaps_per_dir": "100", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_metrics_update_interval": "2", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_min_caps_per_client": "100", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_min_caps_working_set": "10000", 2024-04-24T00:33:50.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_numa_node": "-1", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_complaint_time": "30.000000", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_duration": "600", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_size": "20", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_slow_op_size": "20", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_log_threshold": "5", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_ping_grace": "15", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_ping_interval": "5", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_caps": "30000", 2024-04-24T00:33:50.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_warning_threshold": "262144", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_replay_interval": "1.000000", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_root_ino_gid": "0", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_root_ino_uid": "0", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-24T00:33:50.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_shutdown_check": "0", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_skip_ino": "0", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_max_uid": "4294967294", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_min_uid": "0", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_rstat": "false", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_standby_replay_damaged": "false", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_symlink_recovery": "true", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-24T00:33:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_thrash_exports": "0", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_thrash_fragments": "0", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_tick_interval": "5.000000", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_valgrind_exit": "false", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_verify_backtrace": "1", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_verify_scatter": "false", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_wipe_sessions": "false", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool_debug": "false", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_device_bytes": "1073741824", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_page_set": "false", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_page_size": "65536", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_bytes": "134217728", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_messages": "512", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-24T00:33:50.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_disabled_modules": "", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_max_pg_num_change": "128", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mds_bytes": "134217728", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mds_messages": "128", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mon_bytes": "134217728", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mon_messages": "128", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_osd_bytes": "536870912", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_osd_messages": "8192", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_pool": "true", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-24T00:33:50.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_standby_modules": "true", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_stats_period": "5", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_stats_threshold": "5", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_tick_period": "2", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_allow_pool_delete": "true", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_allow_pool_size_one": "true", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_auth_validate_all_caps": "true", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_bytes": "104857600", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_directed_command_retry": "5", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-24T00:33:50.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_parallel": "3", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_log_interval": "1.000000", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_ping_interval": "10.000000", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_target_rank": "-1", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_file_level": "debug", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_file": "true", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-24T00:33:50.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_journald": "false", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_bootstrap": "false", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_start": "false", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_trim": "true", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cpu_threads": "4", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_crush_min_required_version": "hammer", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_daemon_bytes": "419430400", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-24T00:33:50.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_avail_crit": "5", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_avail_warn": "5", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_size_warn": "16106127360", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_json": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_transactions": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_extra_checks": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_reef": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_squid": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-24T00:33:50.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_down_mkfs_grace": "120", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_down_uptime_grace": "60", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_election_default_strategy": "1", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_election_timeout": "5.000000", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ping_divisor": "2", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_enable_op_tracker": "true", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_fake_pool_delete": "false", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_force_quorum_join": "false", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_globalid_prealloc": "10000", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_detail_to_clog": "false", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_log_update_period": "5", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_max_detail": "50", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog": "true", 2024-04-24T00:33:50.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog_interval": "600", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_host": "172.21.15.64,[v2:172.21.15.64:3301,v1:172.21.15.64:6790],172.21.15.167", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_host_override": "", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_initial_members": "", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease": "5.000000", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-24T00:33:50.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_full_interval": "50", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_max": "10000", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_max_summary": "50", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_log_epochs": "500", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_osd": "10000", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_pg_per_osd": "10000", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_pool_pg_num": "65536", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_force_trim_to": "0", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_skip_sanity": "false", 2024-04-24T00:33:50.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_memory_autotune": "true", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_memory_target": "2147483648", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_beacon_grace": "30", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_digest_period": "5", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_inactive_grace": "60", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_min_osdmap_epochs": "500", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_complaint_time": "30", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_duration": "600", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_size": "20", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_slow_op_size": "20", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-24T00:33:50.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_log_threshold": "5", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_in": "false", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_cache_size": "500", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_down_out_interval": "600", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_force_trim_to": "0", 2024-04-24T00:33:50.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_down_reporters": "2", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-24T00:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_report_timeout": "900", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_stuck_threshold": "60", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-24T00:33:50.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_probe_timeout": "2.000000", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_max_change": "0.050000", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_max_osds": "4", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_interval": "86400", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_max_keys": "100", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_timeout": "300", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_session_timeout": "300", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_smart_report_timeout": "5", 2024-04-24T00:33:50.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stat_smooth_intervals": "6", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_pool_min_size": "2", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_pool_size": "4", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_debug": "false", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_provider_kill_at": "0", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_requester_kill_at": "0", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_timeout": "60.000000", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_target_pg_per_osd": "100", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_tick_interval": "5", 2024-04-24T00:33:50.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_timecheck_interval": "300.000000", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_use_min_delay_socket": "false", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_older_version_delay": "604800", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_misplaced": "false", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_older_version": "true", 2024-04-24T00:33:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_no_app": "false", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": "", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-24T00:33:50.915 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_tracing_enabled": "false", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_op_threads": "3", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_cm": "false", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_device_name": "", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_dscp": "96", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_local_gid": "", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_port_num": "1", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-24T00:33:50.916 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_sl": "3", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_support_srq": "true", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_type": "ib", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_reap_threshold": "5", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_before_connect": "false", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_ipv4": "true", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_ipv6": "false", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_msgr1": "true", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_msgr2": "true", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_port_max": "7568", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_port_min": "6800", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-24T00:33:50.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_retry_count": "3", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_retry_delay": "5", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_client": "false", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mds": "false", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mgr": "false", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mon": "false", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_osd": "false", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_client_mode": "crc secure", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_cluster_mode": "crc secure", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_cluster_type": "", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_compress_secure": "false", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_connection_idle_timeout": "900", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_connection_ready_timeout": "10", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_crc_data": "true", 2024-04-24T00:33:50.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_crc_header": "true", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_bad_msg": "false", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_bug": "true", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_old_message": "true", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_skipped_message": "false", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_coremask": "0xF", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_enable_tso": "true", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hugepages": "", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-24T00:33:50.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_lro": "true", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_memory_channel": "4", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_pmd": "", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_port_id": "0", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dump_on_send": "false", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_initial_backoff": "0.200000", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_max": "1.000000", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_type": "", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_network_congestion": "0", 2024-04-24T00:33:50.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_socket_failures": "5000", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_learn_addr_from_peer": "true", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_max_accept_failures": "4", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_max_backoff": "15.000000", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_client_mode": "secure crc", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_service_mode": "secure crc", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compress_min_size": "1024", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compress_mode": "none", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_pq_min_cost": "65536", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_public_type": "", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_service_mode": "crc secure", 2024-04-24T00:33:50.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_listen_backlog": "512", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_nodelay": "true", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_rcvbuf": "0", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_type": "async+posix", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "no_config_file": "false", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_completion_locks_per_session": "32", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inflight_ops": "1024", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_tick_interval": "5.000000", 2024-04-24T00:33:50.922 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_timeout": "10.000000", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "objectstore_blackhole": "false", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "openssl_engine_opts": "", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_delay_time": "5.000000", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_hist_halflife": "1000", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_max_low_ops": "2", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_max_ops": "4", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_slop": "0.020000", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_asio_thread_count": "2", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_async_recovery_min_cost": "100", 2024-04-24T00:33:50.923 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_scan_max": "512", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_scan_min": "64", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_degraded": "false", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_peering": "false", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_unfound": "true", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_beacon_report_interval": "300", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_duration": "30", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_max_block_size": "67108864", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_blkin_trace_all": "false", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-24T00:33:50.924 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_check_for_log_corruption": "false", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_update_on_start": "true", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_message_cap": "256", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_message_size_cap": "524288000", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_op_priority": "63", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_watch_timeout": "30", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_command_max_records": "256", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_compact_on_start": "false", 2024-04-24T00:33:50.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_update_on_start": "true", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_update_weight_set": "true", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_feed_pullee": "-1", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-24T00:33:50.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_misdirected_ops": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_no_acting_change": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_no_purge_strays": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_op_order": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_shutdown": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_trim_objects": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-24T00:33:50.927 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_snaps": "false", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_keys": "1024", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_stride": "524288", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_default_notify_timeout": "30", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep": "0.000000", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-24T00:33:50.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_discard_disconnected_ops": "true", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_enable_op_tracker": "true", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_info": "true", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown": "true", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_function_tracing": "false", 2024-04-24T00:33:50.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_grace": "20", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_interval": "6", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_peers": "10", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_size": "2000", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_stale": "600", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_max_size": "100000", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_min_size": "1000", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-24T00:33:50.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal_size": "5120", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_kill_backfill_at": "0", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_cache_size": "50", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_dedup": "true", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_message_max": "40", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_share_max_epochs": "40", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_attr_name_len": "100", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_attr_size": "0", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_backfills": "1", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_markdown_count": "5", 2024-04-24T00:33:50.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_markdown_period": "600", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_name_len": "2048", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_namespace_len": "256", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_size": "134217728", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_blocked_by": "16", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_log_entries": "10000", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pgls": "1024", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_push_cost": "8388608", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_push_objects": "10", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_scrubs": "3", 2024-04-24T00:33:50.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_trimming_pgs": "2", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_write_op_reply_len": "64", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_write_size": "90", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_profile": "balanced", 2024-04-24T00:33:50.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_base": "805306368", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_cache_min": "134217728", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-24T00:33:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target": "4294967296", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target_autotune": "false", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_min_pg_log_entries": "250", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_min_recovery_priority": "0", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_report_interval": "5", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_num_cache_shards": "32", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_num_op_tracker_shard": "32", 2024-04-24T00:33:50.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_auto_affinity": "true", 2024-04-24T00:33:50.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_node": "-1", 2024-04-24T00:33:50.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_prefer_iface": "true", 2024-04-24T00:33:50.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-24T00:33:50.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objecter_finishers": "1", 2024-04-24T00:33:50.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore": "bluestore", 2024-04-24T00:33:50.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_fuse": "false", 2024-04-24T00:33:50.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_tracing": "false", 2024-04-24T00:33:50.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_complaint_time": "30.000000", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_duration": "600", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_size": "20", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_slow_op_size": "20", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_log_threshold": "5", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards": "0", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards_hdd": "5", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards_ssd": "8", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_pq_min_cost": "65536", 2024-04-24T00:33:50.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_queue_cut_off": "high", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_thread_timeout": "15", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_open_classes_on_start": "true", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_os_flags": "0", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_peering_op_priority": "255", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_delete_cost": "1048576", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_delete_priority": "5", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_trim_max": "10000", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_trim_min": "100", 2024-04-24T00:33:50.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_crimson": "false", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-24T00:33:50.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flags": "0", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_min_size": "0", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pg_num": "32", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pgp_num": "0", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_read_ratio": "70", 2024-04-24T00:33:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_size": "2", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_type": "replicated", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_push_per_object_cost": "1000", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recover_clone_overlap": "true", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_cost": "20971520", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active": "0", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-24T00:33:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_single_start": "1", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_op_priority": "3", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_priority": "5", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep": "0.000000", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_repair_during_recovery": "false", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_requested_scrub_priority": "120", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-24T00:33:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_auto_repair": "false", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_begin_hour": "0", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_begin_week_day": "0", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_chunk_max": "25", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_chunk_min": "5", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_cost": "52428800", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_disable_reservation_queuing": "true", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_during_recovery": "false", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_end_hour": "0", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_end_week_day": "0", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_event_cost": "4096", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-24T00:33:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_invalid_stats": "true", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_max_preemptions": "5", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_priority": "5", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_sleep": "0.000000", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_skip_data_digest": "false", 2024-04-24T00:33:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_smart_report_timeout": "5", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_cost": "1048576", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_priority": "5", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_target_transaction_size": "30", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-24T00:33:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tracing": "false", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_use_stale_snap": "false", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osdc_blkin_trace_all": "false", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_kill_at": "0", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_max_join_drift": "10", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_min": "500", 2024-04-24T00:33:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_min_wait": "0.050000", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_propose_interval": "1.000000", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_max": "500", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_min": "250", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_stash_full_interval": "25", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_trim_max": "500", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_trim_min": "250", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "perf": "true", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin_dir": "/usr/lib/ceph", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "-", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrv": "", 2024-04-24T00:33:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "public_bind_addr": "-", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "public_network": "", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "public_network_interface": "", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_enabled": "false", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_session_max_number": "256", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_mon_op_timeout": "0", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_osd_op_timeout": "0", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_tracing": "false", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_atime_update_interval": "60", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_balance_parent_reads": "false", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_balance_snap_reads": "false", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blkin_trace_all": "false", 2024-04-24T00:33:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache": "true", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_policy": "writearound", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_size": "33554432", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_clone_copy_on_read": "false", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_compression_hint": "none", 2024-04-24T00:33:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_concurrent_management_ops": "10", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_clone_format": "auto", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_data_pool": "", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_features": "61", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_format": "2", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_map_options": "", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_order": "22", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_pool": "rbd", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_stripe_count": "0", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_stripe_unit": "0", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_enable_alloc_hint": "true", 2024-04-24T00:33:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_io_scheduler": "simple", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_order": "24", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_pool": "", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_splay_width": "4", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_localize_parent_reads": "false", 2024-04-24T00:33:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_localize_snap_reads": "false", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-24T00:33:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-24T00:33:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mtime_update_interval": "60", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_non_blocking_aio": "true", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_op_thread_timeout": "60", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_op_threads": "1", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_parent_cache_enabled": "false", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-24T00:33:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_plugins": "", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_burst": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_limit": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_exclude_ops": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_burst": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_limit": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-24T00:33:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_read_from_replica_policy": "default", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_skip_partial_discard": "true", 2024-04-24T00:33:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_tracing": "false", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_validate_names": "true", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_validate_pool": "true", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "restapi_base_url": "", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "restapi_log_level": "", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_acl_grants_max_num": "100", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_admin_entry": "admin", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_backend_store": "rados", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_barbican_url": "", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_beast_enable_async": "true", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_counters_cache": "false", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-24T00:33:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_enabled": "true", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_expiry_interval": "900", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_lru_size": "10000", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_config_store": "rados", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_content_length_compat": "false", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_copy_obj_progress": "true", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-24T00:33:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cors_rules_max_num": "100", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cross_domain_policy": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_addr": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_password": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_username": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_require_ssl": "true", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-24T00:33:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_addr": "", 2024-04-24T00:33:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_auth": "token", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_namespace": "", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_prefix": "", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_token_file": "", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_buffersize": "524288", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_low_speed_time": "300", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-24T00:33:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_port": "6379", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_changes_size": "1000", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_num_shards": "128", 2024-04-24T00:33:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_window": "30", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-24T00:33:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-24T00:33:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dns_name": "", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dns_s3website_name": "", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dynamic_resharding": "true", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_gc_threads": "true", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_lc_threads": "true", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_ops_log": "false", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_quota_threads": "true", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_static_website": "false", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_usage_log": "false", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enforce_swift_acls": "true", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_exit_timeout_secs": "120", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_expose_bucket": "false", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_extended_http_attrs": "", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_filter": "none", 2024-04-24T00:33:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_frontends": "beast port=7480", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_deferred": "50", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_objs": "32", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_processor_period": "3600", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-24T00:33:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_init_timeout": "300", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_connection_idle": "300", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_domain": "", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_password": "", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_password_path": "", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_project": "", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_tenant": "", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_token": "", 2024-04-24T00:33:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_token_path": "", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_user": "", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_api_version": "2", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_domain": "", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_password": "", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_project": "", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_user": "", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-24T00:33:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_url": "", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_debug_interval": "-1", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_lock_max_time": "90", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_objs": "32", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_rules": "1000", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_worker": "3", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_thread_delay": "0", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_dnattr": "uid", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_searchfilter": "", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-24T00:33:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_uri": "", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_http_headers": "", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_object_name_utc": "false", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attr_name_len": "0", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attr_size": "0", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_chunk_size": "4194304", 2024-04-24T00:33:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_listing_results": "1000", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_notify_retries": "10", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_put_param_size": "1048576", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_put_size": "5368709120", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_slo_entries": "1000", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_md_log_max_shards": "64", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-24T00:33:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_mp_lock_max_time": "600", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_max_gc": "300", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-24T00:33:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_num_async_rados_threads": "32", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_num_control_oids": "8", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_numa_node": "-1", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_chunk_size": "100", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_gc_interval": "600", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-24T00:33:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_thread_timeout": "600", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_tracing": "false", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_token": "", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_url": "", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_verify_ssl": "true", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_rados": "false", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_socket_path": "", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_push_interval": "2.000000", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-24T00:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_lanes": "3", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_partitions": "3", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_print_continue": "true", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_tracing": "false", 2024-04-24T00:33:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm": "", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm_id": "", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_region": "", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_request_uri": "", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_batch_size": "64", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_max_aio": "128", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_num_logs": "16", 2024-04-24T00:33:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_thread_interval": "600", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_resolve_cname": "false", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_run_sync_thread": "true", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_scheduler_type": "throttler", 2024-04-24T00:33:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_script_uri": "", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_service_provider_name": "", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_client_id": "", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_client_secret": "", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_entry": "sts", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_key": "sts", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_min_session_duration": "900", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_token_introspection_url": "", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_account_in_url": "false", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_auth_entry": "auth", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_auth_url": "", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_custom_header": "", 2024-04-24T00:33:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_need_stats": "true", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_tenant_name": "", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_token_expiration": "86400", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_url": "", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_url_prefix": "swift", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_lease_period": "120", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-24T00:33:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_thread_pool_size": "512", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_comment": "", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_createby": "", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_encoding": "", 2024-04-24T00:33:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_flag": "false", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_origin": "", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_tracker": "", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_trust_forwarded_https": "false", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_max_shards": "32", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_max_user_shards": "1", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_use_opa_authz": "false", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_counters_cache": "false", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-24T00:33:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_max_buckets": "1000", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_policies_max_num": "100", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_unique_email": "true", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_verify_ssl": "true", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone": "", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone_id": "", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup": "", 2024-04-24T00:33:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup_id": "", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rgwlc_auto_session_clear": "true", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_block_size": "4096", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_size": "536870912", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-24T00:33:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_index_type": "binary_search", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_partition_filters": "false", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_perf": "false", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "run_dir": "/var/run/ceph", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_block_create": "true", 2024-04-24T00:33:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_cache_lru_size": "67108864", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_default_max_object_size": "16777216", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_device_size": "53687091200", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_capacity": "16", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_main_device_type": "SSD", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-24T00:33:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_segment_size": "67108864", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "set_keepcaps": "false", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "setgroup": "", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "setuser": "", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "setuser_match_path": "", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "thp": "false", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "threadpool_default_timeout": "60", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "throttler_perf_counter": "true" 2024-04-24T00:33:50.988 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:33:50.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-24T00:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a: {} 2024-04-24T00:33:51.132 INFO:teuthology.orchestra.run.smithi064.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T00:33:51.146 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b: {} 2024-04-24T00:33:51.146 INFO:teuthology.orchestra.run.smithi064.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T00:33:51.155 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c: {} 2024-04-24T00:33:51.155 INFO:teuthology.orchestra.run.smithi064.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T00:33:51.171 DEBUG:teuthology.orchestra.run.smithi167:> type ceph-objectstore-tool 2024-04-24T00:33:51.177 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T00:33:51.177 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-04-24T00:33:51.182 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T00:33:51.182 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-04-24T00:33:51.192 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-04-24T00:33:51.192 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-24T00:33:51.192 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:33:51.192 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:33:51.192 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:33:51.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-04-24T00:33:51.193 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-24T00:33:51.193 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-24T00:33:51.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:33:51.194 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-24T00:33:51.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:33:51.195 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-24T00:33:51.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:33:51.196 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 -g pool_client.0 image_client.0 2024-04-24T00:33:51.219 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:51.216+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:51.244 INFO:teuthology.orchestra.run.smithi064.stdout:mapped writes DISABLED 2024-04-24T00:33:51.244 INFO:teuthology.orchestra.run.smithi064.stdout:mapped reads DISABLED 2024-04-24T00:33:51.244 INFO:teuthology.orchestra.run.smithi064.stdout:Seed set to 8129 2024-04-24T00:33:51.296 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:51.292+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:51.369 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:33:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:33:51.370 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:33:51.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:33:51.394 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:33:51.394 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:33:51.397 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:51.392+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:51.411 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:33:51.411 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:33:51.429 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:33:51.430 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:33:51.447 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:33:51.447 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:33:51.467 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:33:51.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:33:51.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:33:51.487 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:33:51.498 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:51.492+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:51.499 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:51.499 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:51.506 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:33:51.506 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:33:51.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:33:51.525 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:33:51.526 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:33:51.526 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:33:51.520+0000 7f841a5f8640 -1 mon.a@0(leader).osd e20 definitely_dead 0 2024-04-24T00:33:51.599 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:51.592+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:51.637 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:51.637 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:51.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:33:51.700 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:51.696+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:51.801 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:51.796+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:51.816 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:51.816 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:51.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:33:51.902 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:51.896+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:51.949 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:51.950 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:51.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:33:52.003 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.5. 2024-04-24T00:33:52.004 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:51.996+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:33:52.015 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:33:52.104 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:52.100+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:52.122 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:52.122 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:52.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:33:52.205 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:52.200+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:33:52.252 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:52.252 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:52.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:33:52.305 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:52.300+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:52.393 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:52.393 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:52.406 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:52.400+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:33:52.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:33:52.507 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:52.500+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:52.568 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:52.569 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:52.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:33:52.629 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:52.624+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:52.709 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:52.704+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:52.710 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:52.711 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:52.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:33:52.813 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:52.808+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:52.894 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:52.895 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:52.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:33:52.910 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:52.904+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:53.010 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:53.004+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:53.095 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:53.095 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:53.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:33:53.111 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:53.104+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:53.212 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:53.208+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:53.245 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:53.245 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:53.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:33:53.347 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:53.344+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:33:53.414 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:53.408+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:53.422 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:53.422 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:53.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:33:53.515 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:53.508+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:53.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:33:53.601 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:53.601 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:53.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:33:53.616 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:53.608+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:53.700 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:33:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:33:53.716 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:53.712+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:53.717 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:33:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:33:53.733 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:33:53.733 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:33:53.741 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:53.741 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:53.750 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:33:53.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:33:53.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:33:53.769 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:33:53.769 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:33:53.789 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:33:53.790 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:33:53.813 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:33:53.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:33:53.817 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:53.812+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:53.832 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:33:53.832 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:33:53.918 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:53.912+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:53.923 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:53.923 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:33:53.923 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:33:53.923 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:33:53.923 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:53.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:33:54.019 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:54.012+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:54.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:33:54.060 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:54.061 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:33:54.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:33:54.061 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [] 2024-04-24T00:33:54.061 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:54.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:33:54.121 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:54.116+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:54.193 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:54.194 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:33:54.194 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:33:54.194 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:54.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:33:54.222 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:54.216+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:33:54.323 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:54.316+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:54.331 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:54.331 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:33:54.331 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:33:54.331 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:33:54.332 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:54.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:33:54.435 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:54.432+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:54.515 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:33:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:33:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [] 2024-04-24T00:33:54.516 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:54.524 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:54.516+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:54.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:33:54.625 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:54.620+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:33:54.694 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:33:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:33:54.694 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:54.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:33:54.726 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:54.724+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:33:54.825 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:54.825 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:33:54.825 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:33:54.825 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:33:54.825 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:54.827 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:54.820+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:54.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:33:54.927 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:54.920+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:55.002 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:55.002 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:33:55.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:33:55.002 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:33:55.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.002 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.002 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.071023746, 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:33:55.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:33:55.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.91248626, 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011455702, 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.009 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 37 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084944+0000", 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1999999999999999e-07 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084950+0000", 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6200000000000001e-06 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084964+0000", 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3186999999999999e-05 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084971+0000", 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3289999999999996e-06 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094143+0000", 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091718640000000001 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094289+0000", 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014654499999999999 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094480+0000", 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019091799999999999 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.095845+0000", 2024-04-24T00:33:55.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00136509 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096272+0000", 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042710300000000002 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096345+0000", 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2243000000000006e-05 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096399+0000", 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4082999999999997e-05 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.617704561, 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013518105000000001, 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.015 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 110 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379726+0000", 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5770000000000001e-06 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379738+0000", 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1552999999999999e-05 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379758+0000", 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9636e-05 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379773+0000", 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4970000000000001e-05 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388655+0000", 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088828440000000009 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388781+0000", 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012547200000000001 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388993+0000", 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021258200000000001 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.392907+0000", 2024-04-24T00:33:55.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039138669999999997 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393039+0000", 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013181900000000001 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393112+0000", 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2853000000000002e-05 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393243+0000", 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130932 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.617633562, 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013665012000000001, 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.021 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 111 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379797+0000", 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.173e-06 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379806+0000", 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3370000000000006e-06 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379825+0000", 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8519999999999999e-05 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379839+0000", 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.401e-05 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390285+0000", 2024-04-24T00:33:55.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010446162 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390412+0000", 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012749899999999999 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390614+0000", 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020195799999999999 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393015+0000", 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024003789999999998 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393268+0000", 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025304800000000002 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393356+0000", 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8613e-05 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393461+0000", 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104313 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.617563056, 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019315533999999999, 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.027 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 112 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379868+0000", 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.274e-06 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380096+0000", 2024-04-24T00:33:55.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022802799999999999 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380146+0000", 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0358999999999997e-05 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380164+0000", 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7713000000000001e-05 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391250+0000", 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011085928 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391383+0000", 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013310899999999999 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391576+0000", 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019287899999999999 2024-04-24T00:33:55.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395703+0000", 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041278740000000001 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398947+0000", 2024-04-24T00:33:55.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032432149999999998 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399035+0000", 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8635999999999996e-05 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399182+0000", 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014651900000000001 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.617217605, 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019210844000000001, 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 113 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.034 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380213+0000", 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5829999999999999e-06 2024-04-24T00:33:55.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380225+0000", 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1603999999999999e-05 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380244+0000", 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9170000000000001e-05 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380258+0000", 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4382999999999999e-05 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393574+0000", 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013315726999999999 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393703+0000", 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012839200000000001 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393904+0000", 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000201805 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398808+0000", 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049032820000000001 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399200+0000", 2024-04-24T00:33:55.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039261 2024-04-24T00:33:55.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399310+0000", 2024-04-24T00:33:55.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399423+0000", 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011273900000000001 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.041 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.617141169, 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019317091000000002, 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 114 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380289+0000", 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.173e-06 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380300+0000", 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-05 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380324+0000", 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3047000000000001e-05 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380338+0000", 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4545999999999999e-05 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394634+0000", 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014296233 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394760+0000", 2024-04-24T00:33:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012605900000000001 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394953+0000", 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019237900000000001 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398924+0000", 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039709589999999996 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399439+0000", 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051536599999999996 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399507+0000", 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8065999999999995e-05 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399605+0000", 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8133000000000004e-05 2024-04-24T00:33:55.047 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.617027707, 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020360119999999999, 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 115 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380404+0000", 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380546+0000", 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014221200000000001 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380561+0000", 2024-04-24T00:33:55.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5696e-05 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380572+0000", 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0353999999999999e-05 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395831+0000", 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015259435 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395982+0000", 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150736 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.396234+0000", 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000252334 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399712+0000", 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034776529999999998 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400554+0000", 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00084231699999999998 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400634+0000", 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9270000000000005e-05 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400762+0000", 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012814900000000001 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.053 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.616837871, 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020559280999999999, 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.054 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 116 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380593+0000", 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380600+0000", 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0160000000000003e-06 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380614+0000", 2024-04-24T00:33:55.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4317e-05 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380622+0000", 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.397613+0000", 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016991532 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.397762+0000", 2024-04-24T00:33:55.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149228 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398030+0000", 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026737499999999998 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400774+0000", 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027444959999999999 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400968+0000", 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193358 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401045+0000", 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7440000000000004e-05 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401151+0000", 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105719 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.616795012000001, 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024520601, 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 117 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.060 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380635+0000", 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:33:55.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380642+0000", 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7700000000000004e-06 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380654+0000", 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1686e-05 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380666+0000", 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2150000000000001e-05 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399787+0000", 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019121461999999999 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399921+0000", 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134043 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400127+0000", 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020607500000000001 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401186+0000", 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001058738 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.404952+0000", 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037661779999999998 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405042+0000", 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9889999999999995e-05 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405155+0000", 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112812 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.108100568999999, 2024-04-24T00:33:55.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013799176999999999, 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 133 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889330+0000", 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.339e-06 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889338+0000", 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9610000000000005e-06 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889352+0000", 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889361+0000", 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8100000000000004e-06 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891442+0000", 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020808860000000001 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891501+0000", 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9314999999999997e-05 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891579+0000", 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7163000000000006e-05 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895799+0000", 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042201670000000004 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903071+0000", 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072725380000000003 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903101+0000", 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9828e-05 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903128+0000", 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6747e-05 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.070 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.108041394000001, 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014011816999999999, 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 134 2024-04-24T00:33:55.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889389+0000", 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3990000000000001e-06 2024-04-24T00:33:55.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889469+0000", 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9915999999999999e-05 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889482+0000", 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3215e-05 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889491+0000", 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8179999999999993e-06 2024-04-24T00:33:55.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891840+0000", 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023490400000000002 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891898+0000", 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8031000000000001e-05 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891977+0000", 2024-04-24T00:33:55.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8512999999999999e-05 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895803+0000", 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00382593 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903233+0000", 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074298970000000004 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903329+0000", 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6718999999999993e-05 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903400+0000", 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0339000000000005e-05 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:33:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.107902279999999, 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013963768999999999, 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889528+0000", 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1100000000000004e-07 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889534+0000", 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3539999999999996e-06 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889545+0000", 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0684e-05 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889553+0000", 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2770000000000005e-06 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892602+0000", 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030488920000000001 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892651+0000", 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8905e-05 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892729+0000", 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896463+0000", 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037338810000000001 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903439+0000", 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069760079999999997 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903468+0000", 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9349e-05 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903491+0000", 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2555999999999999e-05 2024-04-24T00:33:55.081 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.107856459000001, 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01413908, 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.082 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889574+0000", 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1399999999999996e-07 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889579+0000", 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7810000000000002e-06 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889589+0000", 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3889999999999994e-06 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889597+0000", 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7950000000000008e-06 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893012+0000", 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003415408 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893065+0000", 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2932000000000001e-05 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893157+0000", 2024-04-24T00:33:55.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2213999999999995e-05 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896469+0000", 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033114849999999999 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903563+0000", 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070944959999999996 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903649+0000", 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5653000000000001e-05 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903712+0000", 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3113000000000004e-05 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.107797786000001, 2024-04-24T00:33:55.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014166239000000001, 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 137 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889713+0000", 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0840999999999994e-05 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889722+0000", 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7329999999999992e-06 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889728+0000", 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8189999999999997e-06 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893442+0000", 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037142540000000002 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893489+0000", 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7271000000000002e-05 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893566+0000", 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6858000000000002e-05 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896481+0000", 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002914551 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903750+0000", 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072693860000000001 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903776+0000", 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6146999999999999e-05 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903798+0000", 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1478999999999999e-05 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.092 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.107687746, 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015393184000000001, 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 138 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1200000000000003e-07 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889745+0000", 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3759999999999999e-06 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889751+0000", 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2070000000000003e-06 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889755+0000", 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0230000000000003e-06 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894238+0000", 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044834610000000002 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894288+0000", 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9434000000000001e-05 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894364+0000", 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5937000000000002e-05 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896492+0000", 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021286870000000002 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905091+0000", 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085987419999999995 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905116+0000", 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4752e-05 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905135+0000", 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9052999999999999e-05 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.098 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.107662094, 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015493453000000001, 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 139 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.099 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889768+0000", 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889774+0000", 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0050000000000001e-06 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889781+0000", 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7090000000000002e-06 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889785+0000", 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8489999999999999e-06 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894637+0000", 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048518320000000004 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894687+0000", 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0102000000000002e-05 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894765+0000", 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7509000000000006e-05 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896500+0000", 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017353259999999999 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905183+0000", 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086828540000000003 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905208+0000", 2024-04-24T00:33:55.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5622999999999998e-05 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905261+0000", 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2244999999999999e-05 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.107635954999999, 2024-04-24T00:33:55.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018149166000000001, 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889794+0000", 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3200000000000001e-07 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889854+0000", 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0100999999999999e-05 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889861+0000", 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.759e-06 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889865+0000", 2024-04-24T00:33:55.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6059999999999996e-06 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895025+0000", 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051602059999999996 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895081+0000", 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5625999999999999e-05 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895156+0000", 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4684000000000005e-05 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.897187+0000", 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020309880000000001 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907791+0000", 2024-04-24T00:33:55.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010604282 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907877+0000", 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6360000000000007e-05 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907943+0000", 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5222000000000007e-05 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.107554499999999, 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01817011, 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.111 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 141 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2800000000000002e-07 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889879+0000", 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3359999999999999e-06 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889884+0000", 2024-04-24T00:33:55.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.31e-06 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889889+0000", 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8369999999999996e-06 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895829+0000", 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059402509999999997 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895895+0000", 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6210999999999993e-05 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895973+0000", 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7540000000000006e-05 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.897198+0000", 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001225414 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907985+0000", 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010786636 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.908019+0000", 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4258000000000003e-05 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.908045+0000", 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5888999999999999e-05 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.313535400999999, 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011747629000000001, 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 159 2024-04-24T00:33:55.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683895+0000", 2024-04-24T00:33:55.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1299999999999998e-07 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683901+0000", 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.823e-06 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683911+0000", 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.004e-05 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683918+0000", 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.294e-06 2024-04-24T00:33:55.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.692948+0000", 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090303129999999999 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.693199+0000", 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025027200000000001 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.693589+0000", 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039074299999999998 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695297+0000", 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001707854 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695439+0000", 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014188499999999999 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695582+0000", 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000142473 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695642+0000", 2024-04-24T00:33:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0019000000000003e-05 2024-04-24T00:33:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:55.122 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:55.124 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:33:55.124 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:55.116+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:55.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:33:55.224 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:55.220+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:55.289 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:55.289 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:33:55.290 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:33:55.290 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:33:55.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:33:55.326 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:55.320+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:33:55.426 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:55.420+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:33:55.459 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:55.459 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:33:55.459 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:33:55.460 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:33:55.460 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:33:55.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:33:55.527 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:55.524+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:20 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'10, mlcod=16'8)", 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.066304+0000", 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.526557159999999, 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003865393, 2024-04-24T00:33:55.597 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.066304+0000", 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.066304+0000", 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.066306+0000", 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.79e-06 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.066600+0000", 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029444999999999997 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.066630+0000", 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9743999999999998e-05 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.066644+0000", 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3423e-05 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.066788+0000", 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000144079 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.066846+0000", 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8009e-05 2024-04-24T00:33:55.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.069999+0000", 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031530289999999999 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.070170+0000", 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017086900000000001 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:118 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'80, mlcod=16'78)", 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.408124+0000", 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.184737904, 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041389080000000002, 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.408124+0000", 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.408124+0000", 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.408124+0000", 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9999999999999996e-07 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.408376+0000", 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025148599999999998 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.408387+0000", 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1114000000000001e-05 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.408394+0000", 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2439999999999996e-06 2024-04-24T00:33:55.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.408448+0000", 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3804000000000002e-05 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.408464+0000", 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6016999999999999e-05 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412170+0000", 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037061770000000002 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412262+0000", 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2266000000000001e-05 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.607 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:119 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.410217+0000", 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.182644372, 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040812599999999997, 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.410217+0000", 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.410217+0000", 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.410218+0000", 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.09e-07 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.410265+0000", 2024-04-24T00:33:55.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7336999999999998e-05 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.410279+0000", 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4015999999999999e-05 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.410285+0000", 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.139e-06 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.410350+0000", 2024-04-24T00:33:55.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4391000000000001e-05 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.410367+0000", 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7085999999999999e-05 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.414221+0000", 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003853778 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.414298+0000", 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7603999999999998e-05 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:120 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.412029+0000", 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.180832079, 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037820499999999999, 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.612 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412029+0000", 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412029+0000", 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412031+0000", 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.773e-06 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412185+0000", 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015379599999999999 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412195+0000", 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0169e-05 2024-04-24T00:33:55.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412204+0000", 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.439e-06 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412325+0000", 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012152800000000001 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.412364+0000", 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8655999999999997e-05 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.415729+0000", 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033655040000000001 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.415811+0000", 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2185e-05 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:129 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'87, mlcod=16'85)", 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.701746712, 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078162660000000005, 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:33:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891116+0000", 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.747e-06 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891132+0000", 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5282999999999999e-05 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891161+0000", 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9519e-05 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891175+0000", 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891315+0000", 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140032 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891354+0000", 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9199999999999997e-05 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898842+0000", 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074878690000000003 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898931+0000", 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8988999999999998e-05 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:130 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'88, mlcod=16'85)", 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.701057664, 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098350269999999997, 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891805+0000", 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.73e-06 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891819+0000", 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.382e-05 2024-04-24T00:33:55.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891847+0000", 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8155999999999999e-05 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891867+0000", 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9919999999999999e-05 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892020+0000", 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152139 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892101+0000", 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1738999999999998e-05 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901528+0000", 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094262470000000004 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901639+0000", 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111276 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:131 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.700549777999999, 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094484630000000007, 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892313+0000", 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7770000000000001e-06 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892385+0000", 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1939000000000004e-05 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892434+0000", 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9023000000000001e-05 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892449+0000", 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4263e-05 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892995+0000", 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054614900000000001 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893032+0000", 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6863999999999998e-05 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901646+0000", 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086146460000000001 2024-04-24T00:33:55.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901760+0000", 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011380200000000001 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:132 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.699724428, 2024-04-24T00:33:55.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086804629999999994, 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893139+0000", 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.747e-06 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893152+0000", 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.347e-05 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893182+0000", 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9366e-05 2024-04-24T00:33:55.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893198+0000", 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6677e-05 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894622+0000", 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014240489999999999 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894661+0000", 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9019e-05 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901769+0000", 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071080149999999996 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901817+0000", 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.812e-05 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.634 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:133 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'85)", 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.699385117, 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01116346, 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:33:55.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893478+0000", 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3659999999999999e-06 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893490+0000", 2024-04-24T00:33:55.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2784e-05 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893519+0000", 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8983e-05 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893544+0000", 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4813000000000001e-05 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895382+0000", 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018376759999999999 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895433+0000", 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1093e-05 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904576+0000", 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091431289999999998 2024-04-24T00:33:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904640+0000", 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3615999999999995e-05 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:134 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'85)", 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.698965047, 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010781628999999999, 2024-04-24T00:33:55.639 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893898+0000", 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4670000000000001e-06 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893990+0000", 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2221999999999999e-05 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894021+0000", 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0710000000000002e-05 2024-04-24T00:33:55.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894040+0000", 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9303000000000001e-05 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896087+0000", 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020471909999999999 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896120+0000", 2024-04-24T00:33:55.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3056999999999999e-05 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904643+0000", 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085222530000000005 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904678+0000", 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5426000000000003e-05 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:135 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'85)", 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.698162679999999, 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010018802, 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:33:55.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894701+0000", 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.79e-06 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894713+0000", 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2737e-05 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894743+0000", 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0086e-05 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894764+0000", 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1063000000000001e-05 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897677+0000", 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029129609999999999 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897711+0000", 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3763000000000002e-05 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904682+0000", 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069707090000000003 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904718+0000", 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5692999999999999e-05 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:136 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'85)", 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.697781316, 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096742749999999995, 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.649 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895082+0000", 2024-04-24T00:33:55.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.64e-06 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895093+0000", 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895122+0000", 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8989999999999999e-05 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895162+0000", 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9573000000000002e-05 2024-04-24T00:33:55.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898343+0000", 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031815279999999999 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898376+0000", 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2610000000000001e-05 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904722+0000", 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006346338 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904754+0000", 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2033000000000003e-05 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:137 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'85)", 2024-04-24T00:33:55.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.697346606, 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092832910000000008, 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:33:55.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895517+0000", 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895604+0000", 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6876e-05 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895632+0000", 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.834e-05 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895649+0000", 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7266e-05 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.899032+0000", 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033826339999999998 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.899066+0000", 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3603000000000003e-05 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904756+0000", 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056903190000000001 2024-04-24T00:33:55.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904798+0000", 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2132999999999998e-05 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:138 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'96, mlcod=16'85)", 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.696556546, 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010420479, 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.659 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896307+0000", 2024-04-24T00:33:55.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7099999999999999e-06 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896321+0000", 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4847e-05 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896348+0000", 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6216000000000001e-05 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896367+0000", 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.965e-05 2024-04-24T00:33:55.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.900665+0000", 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042981570000000004 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.900727+0000", 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1682999999999999e-05 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906588+0000", 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058611999999999996 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906725+0000", 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137016 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:139 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'97, mlcod=16'85)", 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.696212465, 2024-04-24T00:33:55.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010172130999999999, 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896650+0000", 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.457e-06 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896665+0000", 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5085999999999999e-05 2024-04-24T00:33:55.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896696+0000", 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0505999999999999e-05 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896713+0000", 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7139999999999999e-05 2024-04-24T00:33:55.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901935+0000", 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052216169999999996 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901974+0000", 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9632000000000003e-05 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906732+0000", 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047579370000000003 2024-04-24T00:33:55.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906821+0000", 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8756000000000002e-05 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:140 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'85)", 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.695713199, 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012183782000000001, 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897149+0000", 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.147e-06 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897223+0000", 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4116000000000007e-05 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897251+0000", 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7606e-05 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897269+0000", 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7476999999999999e-05 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902553+0000", 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052847359999999999 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902586+0000", 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2802000000000002e-05 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909235+0000", 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066493519999999999 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909332+0000", 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6545999999999993e-05 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'85)", 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.694903089, 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011427935, 2024-04-24T00:33:55.672 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897960+0000", 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1500000000000002e-06 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897974+0000", 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3890000000000001e-05 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898000+0000", 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.599e-05 2024-04-24T00:33:55.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898018+0000", 2024-04-24T00:33:55.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7903000000000001e-05 2024-04-24T00:33:55.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903913+0000", 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058942839999999996 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903976+0000", 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3847e-05 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909338+0000", 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053611850000000001 2024-04-24T00:33:55.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909386+0000", 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8686000000000001e-05 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:143 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'101, mlcod=16'98)", 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.911865+0000", 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.680996507, 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004471109, 2024-04-24T00:33:55.677 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.911865+0000", 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.911865+0000", 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.911867+0000", 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.911934+0000", 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7349999999999997e-05 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.911969+0000", 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4906000000000001e-05 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.911993+0000", 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4337000000000001e-05 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.912482+0000", 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048868299999999996 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.912542+0000", 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.916222+0000", 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036802340000000001 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.916336+0000", 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011409299999999999 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:144 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'102, mlcod=16'98)", 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.913466+0000", 2024-04-24T00:33:55.681 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.679395756, 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044350489999999999, 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.913466+0000", 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.913466+0000", 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.913467+0000", 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7540000000000001e-06 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.913522+0000", 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4768999999999997e-05 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.913548+0000", 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5769999999999999e-05 2024-04-24T00:33:55.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.913560+0000", 2024-04-24T00:33:55.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1626e-05 2024-04-24T00:33:55.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.914327+0000", 2024-04-24T00:33:55.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076698100000000002 2024-04-24T00:33:55.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.914368+0000", 2024-04-24T00:33:55.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1056999999999997e-05 2024-04-24T00:33:55.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.917791+0000", 2024-04-24T00:33:55.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00342334 2024-04-24T00:33:55.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.917901+0000", 2024-04-24T00:33:55.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109752 2024-04-24T00:33:55.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.688 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.689 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-24T00:33:55.689 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:33:55.689 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.56044342000000003, 2024-04-24T00:33:55.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097809650000000008, 2024-04-24T00:33:55.689 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:55.689 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:55.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:55.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:33:55.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:33:55.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:55.690 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 11 2024-04-24T00:33:55.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.690 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:55.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:55.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:33:55.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:55.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:33:55.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:55.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:55.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032421+0000", 2024-04-24T00:33:55.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3730000000000001e-06 2024-04-24T00:33:55.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032435+0000", 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3468999999999999e-05 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032461+0000", 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:55.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032473+0000", 2024-04-24T00:33:55.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1613999999999999e-05 2024-04-24T00:33:55.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032624+0000", 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015161999999999999 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032631+0000", 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8700000000000003e-06 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037449+0000", 2024-04-24T00:33:55.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048180180000000003 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037551+0000", 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010177 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037724+0000", 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172943 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:55.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039447+0000", 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017229560000000001 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042096+0000", 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002649069 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042164+0000", 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7931999999999999e-05 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:55.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042199+0000", 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123000000000002e-05 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:55.698 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:33:55.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:33:55.700 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:55.692+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:55.800 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:55.796+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:55.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:33:55.857 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:55.857 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:33:55.857 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:33:55.857 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:33:55.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:33:55.901 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:55.896+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:55.987 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:33:55.988 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:33:56.002 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:55.996+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:56.005 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:33:56.005 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:33:56.021 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:33:56.021 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:33:56.037 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:56.037 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:56.039 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:33:56.040 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:33:56.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:33:56.060 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:33:56.060 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:33:56.080 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:33:56.081 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:33:56.103 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:56.100+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:56.108 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:33:56.108 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:33:56.129 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:33:56.129 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:33:56.168 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:56.168 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:56.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:33:56.205 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:56.200+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:56.305 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:56.300+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:56.336 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:56.337 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:56.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:33:56.406 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:56.400+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:56.468 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:56.468 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:56.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:33:56.525 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:56.520+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:56.609 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:56.604+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:56.643 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:56.643 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:56.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:33:56.744 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:56.740+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:56.810 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:56.804+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:56.825 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:56.825 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:56.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:33:56.924 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:56.920+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:56.980 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0x2a97805 2024-04-24T00:33:56.982 INFO:teuthology.orchestra.run.smithi064.stdout:1 trunc from 0x0 to 0x2a97805 2024-04-24T00:33:56.994 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:56.994 INFO:teuthology.orchestra.run.smithi064.stdout:4 read 0x62ff10 thru 0x63eb6b (0xec5c bytes) 2024-04-24T00:33:56.994 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0x74905de 2024-04-24T00:33:56.999 INFO:teuthology.orchestra.run.smithi064.stdout:5 trunc from 0x2a97805 to 0x74905de 2024-04-24T00:33:57.012 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:33:57.008+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:33:57.014 INFO:teuthology.orchestra.run.smithi064.stdout:6 read 0x3b42f22 thru 0x3b4a6a7 (0x7786 bytes) 2024-04-24T00:33:57.014 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:57.015 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:57.015 INFO:teuthology.orchestra.run.smithi064.stdout:7 read 0x4951054 thru 0x49593fa (0x83a7 bytes) 2024-04-24T00:33:57.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:33:57.033 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:33:57.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T00:33:57.035 INFO:teuthology.orchestra.run.smithi064.stdout:8 writesame 0xbd1fa80 thru 0xbd2d822 data_size 0x1(0xdda3 bytes) 2024-04-24T00:33:57.035 INFO:teuthology.orchestra.run.smithi064.stdout:10 read 0x30d2c4a thru 0x30d2ca8 (0x5f bytes) 2024-04-24T00:33:57.035 INFO:teuthology.orchestra.run.smithi064.stdout:11 read 0x38ea157 thru 0x38f1aa6 (0x7950 bytes) 2024-04-24T00:33:57.035 INFO:teuthology.orchestra.run.smithi064.stdout:12 read 0x106651d thru 0x10690a6 (0x2b8a bytes) 2024-04-24T00:33:57.051 INFO:teuthology.orchestra.run.smithi064.stdout:13 write 0xd107318 thru 0xd116aef (0xf7d8 bytes) 2024-04-24T00:33:57.068 INFO:teuthology.orchestra.run.smithi064.stdout:14 trunc from 0xd116af0 to 0x485e934 2024-04-24T00:33:57.089 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.106 INFO:teuthology.orchestra.run.smithi064.stdout:18 write 0x955149d thru 0x9551f63 (0xac7 bytes) 2024-04-24T00:33:57.112 INFO:teuthology.orchestra.run.smithi064.stdout:19 read 0x6fa0f33 thru 0x6fa8444 (0x7512 bytes) 2024-04-24T00:33:57.112 INFO:teuthology.orchestra.run.smithi064.stdout:20 read 0x57ef6fa thru 0x57f3182 (0x3a89 bytes) 2024-04-24T00:33:57.112 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:57.104+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:57.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:33:57.127 INFO:teuthology.orchestra.run.smithi064.stdout:21 writesame 0xc175154 thru 0xc17f2a8 data_size 0x27(0xa155 bytes) 2024-04-24T00:33:57.127 INFO:teuthology.orchestra.run.smithi064.stdout:22 writesame 0x11a34d0 thru 0x11a5574 data_size 0x1(0x20a5 bytes) 2024-04-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi064.stdout:23 punch from 0x9b23a88 to 0x9b295a5, (0x5b1d bytes) 2024-04-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi064.stdout:24 punch from 0x6d6d294 to 0x6d71bce, (0x493a bytes) 2024-04-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi064.stdout:25 write 0x57a6374 thru 0x57b0ada (0xa767 bytes) 2024-04-24T00:33:57.150 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.150 INFO:teuthology.orchestra.run.smithi064.stdout:27 write 0x84f4c44 thru 0x8500622 (0xb9df bytes) 2024-04-24T00:33:57.156 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:57.156 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:57.163 INFO:teuthology.orchestra.run.smithi064.stdout:29 read 0x89fdc78 thru 0x8a0baad (0xde36 bytes) 2024-04-24T00:33:57.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:33:57.176 INFO:teuthology.orchestra.run.smithi064.stdout:31 write 0xdc30c9f thru 0xdc3583f (0x4ba1 bytes) 2024-04-24T00:33:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:34 read 0x6acfd11 thru 0x6ad01b4 (0x4a4 bytes) 2024-04-24T00:33:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:35 write 0xaa22bc1 thru 0xaa23065 (0x4a5 bytes) 2024-04-24T00:33:57.187 INFO:teuthology.orchestra.run.smithi064.stdout:36 write 0x6c08d40 thru 0x6c11d78 (0x9039 bytes) 2024-04-24T00:33:57.193 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.193 INFO:teuthology.orchestra.run.smithi064.stdout:38 writesame 0x21a3585 thru 0x21b1b16 data_size 0x1e(0xe592 bytes) 2024-04-24T00:33:57.201 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0xa929191 2024-04-24T00:33:57.206 INFO:teuthology.orchestra.run.smithi064.stdout:39 trunc from 0xdc35840 to 0xa929191 2024-04-24T00:33:57.219 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:57.216+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:57.222 INFO:teuthology.orchestra.run.smithi064.stdout:40 punch from 0x3afaeaa to 0x3afcff9, (0x214f bytes) 2024-04-24T00:33:57.235 INFO:teuthology.orchestra.run.smithi064.stdout:42 writesame 0xeb0d82b thru 0xeb12849 data_size 0x9(0x501f bytes) 2024-04-24T00:33:57.236 INFO:teuthology.orchestra.run.smithi064.stdout:43 read 0x5683f2 thru 0x56fa2a (0x7639 bytes) 2024-04-24T00:33:57.236 INFO:teuthology.orchestra.run.smithi064.stdout:44 write 0x1d3238b thru 0x1d340bc (0x1d32 bytes) 2024-04-24T00:33:57.239 INFO:teuthology.orchestra.run.smithi064.stdout:46 writesame 0xbed612a thru 0xbeddb4d data_size 0x2(0x7a24 bytes) 2024-04-24T00:33:57.281 INFO:teuthology.orchestra.run.smithi064.stdout:47 read 0xc88b12c thru 0xc890b04 (0x59d9 bytes) 2024-04-24T00:33:57.281 INFO:teuthology.orchestra.run.smithi064.stdout:48 punch from 0x11b6f12 to 0x11c3dd9, (0xcec7 bytes) 2024-04-24T00:33:57.281 INFO:teuthology.orchestra.run.smithi064.stdout:50 read 0x24991d3 thru 0x249c006 (0x2e34 bytes) 2024-04-24T00:33:57.282 INFO:teuthology.orchestra.run.smithi064.stdout:51 write 0x25dd612 thru 0x25ed523 (0xff12 bytes) 2024-04-24T00:33:57.289 INFO:teuthology.orchestra.run.smithi064.stdout:52 read 0x4b9d08c thru 0x4ba8e8e (0xbe03 bytes) 2024-04-24T00:33:57.289 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.289 INFO:teuthology.orchestra.run.smithi064.stdout:56 write 0x289bbb2 thru 0x28aa82c (0xec7b bytes) 2024-04-24T00:33:57.297 INFO:teuthology.orchestra.run.smithi064.stdout:57 write 0xb1c17ac thru 0xb1c2ee5 (0x173a bytes) 2024-04-24T00:33:57.301 INFO:teuthology.orchestra.run.smithi064.stdout:59 read 0xab0b330 thru 0xab17e58 (0xcb29 bytes) 2024-04-24T00:33:57.301 INFO:teuthology.orchestra.run.smithi064.stdout:60 writesame 0x8d52c64 thru 0x8d54e28 data_size 0x13(0x21c5 bytes) 2024-04-24T00:33:57.306 INFO:teuthology.orchestra.run.smithi064.stdout:61 read 0x9514c1f thru 0x951b30f (0x66f1 bytes) 2024-04-24T00:33:57.306 INFO:teuthology.orchestra.run.smithi064.stdout:62 write 0xb4763fb thru 0xb47fafa (0x9700 bytes) 2024-04-24T00:33:57.313 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.314 INFO:teuthology.orchestra.run.smithi064.stdout:64 punch from 0x38990 to 0x47725, (0xed95 bytes) 2024-04-24T00:33:57.314 INFO:teuthology.orchestra.run.smithi064.stdout:65 punch from 0x1f625ee to 0x1f640d6, (0x1ae8 bytes) 2024-04-24T00:33:57.314 INFO:teuthology.orchestra.run.smithi064.stdout:67 read 0xdd8e238 thru 0xdd9391f (0x56e8 bytes) 2024-04-24T00:33:57.314 INFO:teuthology.orchestra.run.smithi064.stdout:71 read 0xd0d0f00 thru 0xd0e0ae7 (0xfbe8 bytes) 2024-04-24T00:33:57.314 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:57.308+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:33:57.314 INFO:teuthology.orchestra.run.smithi064.stdout:72 write 0x76994d0 thru 0x76a013a (0x6c6b bytes) 2024-04-24T00:33:57.319 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:57.319 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:57.332 INFO:teuthology.orchestra.run.smithi064.stdout:73 write 0xecdc0a9 thru 0xece9bce (0xdb26 bytes) 2024-04-24T00:33:57.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:33:57.338 INFO:teuthology.orchestra.run.smithi064.stdout:75 read 0xa5d5900 thru 0xa5e43e7 (0xeae8 bytes) 2024-04-24T00:33:57.338 INFO:teuthology.orchestra.run.smithi064.stdout:76 write 0x4d70782 thru 0x4d71997 (0x1216 bytes) 2024-04-24T00:33:57.342 INFO:teuthology.orchestra.run.smithi064.stdout:77 write 0xb30928e thru 0xb30c7eb (0x355e bytes) 2024-04-24T00:33:57.344 INFO:teuthology.orchestra.run.smithi064.stdout:78 write 0x8455ee7 thru 0x8460596 (0xa6b0 bytes) 2024-04-24T00:33:57.349 INFO:teuthology.orchestra.run.smithi064.stdout:79 read 0x600ec7e thru 0x601ab82 (0xbf05 bytes) 2024-04-24T00:33:57.349 INFO:teuthology.orchestra.run.smithi064.stdout:80 read 0xe02eac2 thru 0xe03a918 (0xbe57 bytes) 2024-04-24T00:33:57.349 INFO:teuthology.orchestra.run.smithi064.stdout:81 read 0x701a3d6 thru 0x7029ab8 (0xf6e3 bytes) 2024-04-24T00:33:57.349 INFO:teuthology.orchestra.run.smithi064.stdout:83 read 0x2316bc0 thru 0x231f297 (0x86d8 bytes) 2024-04-24T00:33:57.350 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.356 INFO:teuthology.orchestra.run.smithi064.stdout:85 trunc from 0xece9bcf to 0xa04304f 2024-04-24T00:33:57.377 INFO:teuthology.orchestra.run.smithi064.stdout:86 trunc from 0xa04304f to 0x60cc80f 2024-04-24T00:33:57.414 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:57.408+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:57.415 INFO:teuthology.orchestra.run.smithi064.stdout:87 write 0xc9d7ec5 thru 0xc9e725a (0xf396 bytes) 2024-04-24T00:33:57.430 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T00:33:57.434 INFO:teuthology.orchestra.run.smithi064.stdout:88 trunc from 0xc9e725b to 0x2fb4f28 2024-04-24T00:33:57.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T00:33:57.478 INFO:teuthology.orchestra.run.smithi064.stdout:89 write 0xe374e67 thru 0xe379e79 (0x5013 bytes) 2024-04-24T00:33:57.481 INFO:teuthology.orchestra.run.smithi064.stdout:90 writesame 0x22148c3 thru 0x221c899 data_size 0x1(0x7fd7 bytes) 2024-04-24T00:33:57.512 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:57.512 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:57.515 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:57.512+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:57.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:33:57.554 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.554 INFO:teuthology.orchestra.run.smithi064.stdout:92 write 0x4ec0760 thru 0x4ecca55 (0xc2f6 bytes) 2024-04-24T00:33:57.566 INFO:teuthology.orchestra.run.smithi064.stdout:93 read 0x9f43c0d thru 0x9f5228c (0xe680 bytes) 2024-04-24T00:33:57.566 INFO:teuthology.orchestra.run.smithi064.stdout:94 read 0xcfda0eb thru 0xcfe7cfb (0xdc11 bytes) 2024-04-24T00:33:57.566 INFO:teuthology.orchestra.run.smithi064.stdout:95 write 0x9d7036e thru 0x9d75722 (0x53b5 bytes) 2024-04-24T00:33:57.571 INFO:teuthology.orchestra.run.smithi064.stdout:96 writesame 0x3140399 thru 0x31415f4 data_size 0xa(0x125c bytes) 2024-04-24T00:33:57.576 INFO:teuthology.orchestra.run.smithi064.stdout:97 writesame 0xa08a6e1 thru 0xa08d5b2 data_size 0x2(0x2ed2 bytes) 2024-04-24T00:33:57.591 INFO:teuthology.orchestra.run.smithi064.stdout:98 write 0x6e9bfab thru 0x6ea6065 (0xa0bb bytes) 2024-04-24T00:33:57.598 INFO:teuthology.orchestra.run.smithi064.stdout:101 read 0xd5ff4b9 thru 0xd60deac (0xe9f4 bytes) 2024-04-24T00:33:57.598 INFO:teuthology.orchestra.run.smithi064.stdout:102 read 0xc5073e0 thru 0xc510a04 (0x9625 bytes) 2024-04-24T00:33:57.614 INFO:teuthology.orchestra.run.smithi064.stdout:103 trunc from 0xe379e7a to 0x3134b45 2024-04-24T00:33:57.616 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:57.612+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:33:57.630 INFO:teuthology.orchestra.run.smithi064.stdout:105 writesame 0x1941cd9 thru 0x194e5de data_size 0x1b(0xc906 bytes) 2024-04-24T00:33:57.657 INFO:teuthology.orchestra.run.smithi064.stdout:106 writesame 0xb47cbf6 thru 0xb480991 data_size 0x2(0x3d9c bytes) 2024-04-24T00:33:57.658 INFO:teuthology.orchestra.run.smithi064.stdout:107 read 0x3624432 thru 0x362f091 (0xac60 bytes) 2024-04-24T00:33:57.658 INFO:teuthology.orchestra.run.smithi064.stdout:108 writesame 0x9f23651 thru 0x9f31348 data_size 0xc(0xdcf8 bytes) 2024-04-24T00:33:57.669 INFO:teuthology.orchestra.run.smithi064.stdout:110 trunc from 0xb480992 to 0x8a83e32 2024-04-24T00:33:57.685 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0xce73078 2024-04-24T00:33:57.692 INFO:teuthology.orchestra.run.smithi064.stdout:111 trunc from 0x8a83e32 to 0xce73078 2024-04-24T00:33:57.693 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:57.693 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:57.703 INFO:teuthology.orchestra.run.smithi064.stdout:113 writesame 0xb72a60 thru 0xb72e04 data_size 0x1(0x3a5 bytes) 2024-04-24T00:33:57.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:33:57.729 INFO:teuthology.orchestra.run.smithi064.stdout:114 trunc from 0xce73078 to 0x43ab900 2024-04-24T00:33:57.755 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:57.752+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:57.765 INFO:teuthology.orchestra.run.smithi064.stdout:115 write 0xcd5680b thru 0xcd66597 (0xfd8d bytes) 2024-04-24T00:33:57.772 INFO:teuthology.orchestra.run.smithi064.stdout:116 write 0x45b4cc2 thru 0x45bc75f (0x7a9e bytes) 2024-04-24T00:33:57.783 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.783 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:33:57.783 INFO:teuthology.orchestra.run.smithi064.stdout:120 read 0x44a2873 thru 0x44adc49 (0xb3d7 bytes) 2024-04-24T00:33:57.784 INFO:teuthology.orchestra.run.smithi064.stdout:121 write 0xad57e15 thru 0xad59819 (0x1a05 bytes) 2024-04-24T00:33:57.790 INFO:teuthology.orchestra.run.smithi064.stdout:122 write 0x4036886 thru 0x40466fa (0xfe75 bytes) 2024-04-24T00:33:57.803 INFO:teuthology.orchestra.run.smithi064.stdout:123 read 0x4597f98 thru 0x45a1605 (0x966e bytes) 2024-04-24T00:33:57.804 INFO:teuthology.orchestra.run.smithi064.stdout:125 punch from 0x2d651a to 0x2de357, (0x7e3d bytes) 2024-04-24T00:33:57.804 INFO:teuthology.orchestra.run.smithi064.stdout:126 write 0x883c0d6 thru 0x884466b (0x8596 bytes) 2024-04-24T00:33:57.809 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0xdf4cf90 2024-04-24T00:33:57.811 INFO:teuthology.orchestra.run.smithi064.stdout:128 trunc from 0xcd66598 to 0xdf4cf90 2024-04-24T00:33:57.818 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:57.812+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:57.836 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:57.836 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:57.836 INFO:teuthology.orchestra.run.smithi064.stdout:129 trunc from 0xdf4cf90 to 0x55e29 2024-04-24T00:33:57.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:33:57.860 INFO:teuthology.orchestra.run.smithi064.stdout:130 punch from 0x17171 to 0x26635, (0xf4c4 bytes) 2024-04-24T00:33:57.883 INFO:teuthology.orchestra.run.smithi064.stdout:131 write 0x9024ed5 thru 0x902ca6d (0x7b99 bytes) 2024-04-24T00:33:57.889 INFO:teuthology.orchestra.run.smithi064.stdout:133 write 0x1a4568c thru 0x1a50684 (0xaff9 bytes) 2024-04-24T00:33:57.901 INFO:teuthology.orchestra.run.smithi064.stdout:134 write 0x1a05946 thru 0x1a102cb (0xa986 bytes) 2024-04-24T00:33:57.908 INFO:teuthology.orchestra.run.smithi064.stdout:135 punch from 0x5636de6 to 0x5638ccf, (0x1ee9 bytes) 2024-04-24T00:33:57.909 INFO:teuthology.orchestra.run.smithi064.stdout:136 read 0x7e29f94 thru 0x7e32681 (0x86ee bytes) 2024-04-24T00:33:57.911 INFO:teuthology.orchestra.run.smithi064.stdout:139 trunc from 0x902ca6e to 0x8030646 2024-04-24T00:33:57.919 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:57.912+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:57.929 INFO:teuthology.orchestra.run.smithi064.stdout:140 read 0x88c1ab thru 0x88fd11 (0x3b67 bytes) 2024-04-24T00:33:57.951 INFO:teuthology.orchestra.run.smithi064.stdout:141 write 0xc978dc7 thru 0xc98615a (0xd394 bytes) 2024-04-24T00:33:57.958 INFO:teuthology.orchestra.run.smithi064.stdout:143 read 0x2aeb242 thru 0x2aee742 (0x3501 bytes) 2024-04-24T00:33:57.958 INFO:teuthology.orchestra.run.smithi064.stdout:145 read 0x6c46113 thru 0x6c489f9 (0x28e7 bytes) 2024-04-24T00:33:57.958 INFO:teuthology.orchestra.run.smithi064.stdout:147 write 0xa9a6f7e thru 0xa9a8b3b (0x1bbe bytes) 2024-04-24T00:33:57.982 INFO:teuthology.orchestra.run.smithi064.stdout:148 writesame 0xe4723a7 thru 0xe47d543 data_size 0x1(0xb19d bytes) 2024-04-24T00:33:57.982 INFO:teuthology.orchestra.run.smithi064.stdout:149 read 0x187820f thru 0x187e8b1 (0x66a3 bytes) 2024-04-24T00:33:57.983 INFO:teuthology.orchestra.run.smithi064.stdout:150 write 0x5a70070 thru 0x5a71b59 (0x1aea bytes) 2024-04-24T00:33:57.990 INFO:teuthology.orchestra.run.smithi064.stdout:151 writesame 0x83c8797 thru 0x83d447d data_size 0x1(0xbce7 bytes) 2024-04-24T00:33:58.020 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:58.016+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:58.022 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:58.022 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:58.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:33:58.121 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:58.116+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:58.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:33:58.148 INFO:teuthology.orchestra.run.smithi064.stdout:152 write 0x76218fb thru 0x7627737 (0x5e3d bytes) 2024-04-24T00:33:58.194 INFO:teuthology.orchestra.run.smithi064.stdout:153 trunc from 0xe47d544 to 0x2c8b9 2024-04-24T00:33:58.205 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:58.205 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:58.219 INFO:teuthology.orchestra.run.smithi064.stdout:154 read 0x10314 thru 0x1784f (0x753c bytes) 2024-04-24T00:33:58.219 INFO:teuthology.orchestra.run.smithi064.stdout:155 read 0x16a87 thru 0x204f7 (0x9a71 bytes) 2024-04-24T00:33:58.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:33:58.221 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:58.216+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:58.233 INFO:teuthology.orchestra.run.smithi064.stdout:159 write 0x12d9a56 thru 0x12e2512 (0x8abd bytes) 2024-04-24T00:33:58.238 INFO:teuthology.orchestra.run.smithi064.stdout:161 read 0xaec44e thru 0xaed84f (0x1402 bytes) 2024-04-24T00:33:58.238 INFO:teuthology.orchestra.run.smithi064.stdout:162 punch from 0xd5041c to 0xd51870, (0x1454 bytes) 2024-04-24T00:33:58.238 INFO:teuthology.orchestra.run.smithi064.stdout:163 read 0xb69942 thru 0xb6ef68 (0x5627 bytes) 2024-04-24T00:33:58.244 INFO:teuthology.orchestra.run.smithi064.stdout:164 trunc from 0x12e2513 to 0x3e66185 2024-04-24T00:33:58.253 INFO:teuthology.orchestra.run.smithi064.stdout:166 punch from 0x1a54036 to 0x1a5c617, (0x85e1 bytes) 2024-04-24T00:33:58.256 INFO:teuthology.orchestra.run.smithi064.stdout:168 trunc from 0x3e66185 to 0x56e2079 2024-04-24T00:33:58.284 INFO:teuthology.orchestra.run.smithi064.stdout:170 write 0xa6a222c thru 0xa6a8553 (0x6328 bytes) 2024-04-24T00:33:58.288 INFO:teuthology.orchestra.run.smithi064.stdout:171 punch from 0x1f8435d to 0x1f8ddb5, (0x9a58 bytes) 2024-04-24T00:33:58.289 INFO:teuthology.orchestra.run.smithi064.stdout:172 clone 1 order 24 su 1048576 sc 7 2024-04-24T00:33:58.308 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:33:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:33:58.322 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:58.316+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:58.325 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:33:58.325 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:33:58.341 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:33:58.341 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:33:58.347 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:33:58.347 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:33:58.357 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:33:58.357 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:33:58.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:33:58.376 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:33:58.376 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:33:58.395 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:33:58.396 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:33:58.415 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:33:58.415 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:33:58.422 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:58.416+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:58.433 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:33:58.433 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:33:58.439 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:33:58.445 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T00:33:58.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T00:33:58.490 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:58.490 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:33:58.490 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:33:58.490 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:33:58.490 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:58.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:33:58.547 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:33:58.544+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:33:58.624 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:58.620+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:13 3.c e24/21 3:32b095e4:::rbd_object_map.115e92aeb10c:head v 24'1, mlcod=0'0)", 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:55.049325+0000", 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6178044370000002, 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000905758, 2024-04-24T00:33:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.049325+0000", 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.049325+0000", 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.049325+0000", 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7099999999999998e-07 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.049329+0000", 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.267e-06 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.049339+0000", 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9930000000000007e-06 2024-04-24T00:33:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.049347+0000", 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3159999999999999e-06 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.049412+0000", 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5136999999999997e-05 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.049438+0000", 2024-04-24T00:33:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6789999999999999e-05 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.050183+0000", 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00074460699999999999 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:55.050230+0000", 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7376999999999997e-05 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:526 3.c e26/21 3:32b095e4:::rbd_object_map.115e92aeb10c:head v 26'106, mlcod=25'104)", 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.453317+0000", 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.21381175799999999, 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000848089, 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.453317+0000", 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.453317+0000", 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.453318+0000", 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3999999999999997e-07 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.453324+0000", 2024-04-24T00:33:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3010000000000003e-06 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.453341+0000", 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6696000000000001e-05 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.453350+0000", 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4490000000000005e-06 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.453404+0000", 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3582999999999998e-05 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.453420+0000", 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6337999999999998e-05 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.454104+0000", 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068396500000000003 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.454165+0000", 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1416999999999998e-05 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:546 3.c e26/21 3:32b095e4:::rbd_object_map.115e92aeb10c:head v 26'107, mlcod=25'105)", 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.488390+0000", 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.17873956699999999, 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001367399, 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.488390+0000", 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.488390+0000", 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.488391+0000", 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.361e-06 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.488401+0000", 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0142e-05 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.488420+0000", 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8751999999999999e-05 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.488432+0000", 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1958e-05 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.488557+0000", 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125535 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.488599+0000", 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2033000000000002e-05 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.489676+0000", 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010764150000000001 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.489757+0000", 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1203000000000002e-05 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:556 3.2 e26/21 3:4550e1e5:::rbd_id.image_client.0-clone1:head v 26'4, mlcod=25'2)", 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.500814+0000", 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.16631522100000001, 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015088160000000001, 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.500814+0000", 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.500814+0000", 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.500815+0000", 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.474e-06 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.500826+0000", 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0886e-05 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.500845+0000", 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8929000000000001e-05 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.500854+0000", 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0860000000000008e-06 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.500949+0000", 2024-04-24T00:33:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4325000000000002e-05 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.500987+0000", 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8535999999999998e-05 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502202+0000", 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012143639999999999 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502323+0000", 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121216 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:557 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.502655+0000", 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.16447439799999999, 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024169009999999999, 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 557 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502655+0000", 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502655+0000", 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502656+0000", 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.313e-06 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502662+0000", 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5300000000000004e-06 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502673+0000", 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.169e-05 2024-04-24T00:33:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502679+0000", 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7880000000000003e-06 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502782+0000", 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102835 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.503009+0000", 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022671799999999999 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.503357+0000", 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000348334 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.504504+0000", 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011466409999999999 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.504870+0000", 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036630099999999997 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.504957+0000", 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7496000000000004e-05 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505072+0000", 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114255 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:558 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.set_stripe_unit_count in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.505546+0000", 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.161582647, 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018824619999999999, 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 558 2024-04-24T00:33:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505546+0000", 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505546+0000", 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505549+0000", 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505560+0000", 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1362999999999999e-05 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505589+0000", 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8649999999999998e-05 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505603+0000", 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4633e-05 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505710+0000", 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106789 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505862+0000", 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015171600000000001 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.506310+0000", 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044782699999999998 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.507099+0000", 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00078961700000000005 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.507235+0000", 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013620599999999999 2024-04-24T00:33:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.507320+0000", 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4828999999999997e-05 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.507429+0000", 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010863500000000001 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:562 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.513675+0000", 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.153454173, 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001132681, 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 562 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.513675+0000", 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.513675+0000", 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.513677+0000", 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.057e-06 2024-04-24T00:33:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.513688+0000", 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.081e-05 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.513716+0000", 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8447000000000001e-05 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.513732+0000", 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6022999999999999e-05 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.513884+0000", 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152008 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.514141+0000", 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025621800000000001 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.514808+0000", 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066711799999999996 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:563 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.515129+0000", 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.15199964999999999, 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00091382599999999996, 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 563 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.515129+0000", 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.515129+0000", 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.515132+0000", 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.03e-06 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.515144+0000", 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.289e-05 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.515173+0000", 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9009e-05 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.515190+0000", 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6120000000000002e-05 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.515316+0000", 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000126095 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.515544+0000", 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000228106 2024-04-24T00:33:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516043+0000", 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00049957599999999995 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:564 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.516220+0000", 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.150909139, 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048189000000000001, 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 564 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516220+0000", 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516220+0000", 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516222+0000", 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7230000000000001e-06 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516229+0000", 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.43e-06 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516248+0000", 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9046999999999999e-05 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516259+0000", 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0373e-05 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516356+0000", 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7751999999999995e-05 2024-04-24T00:33:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516494+0000", 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013742299999999999 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516702+0000", 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000208142 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:565 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call,call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.516908+0000", 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.150221471, 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00086288599999999999, 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 565 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516908+0000", 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516908+0000", 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516909+0000", 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7069999999999999e-06 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516918+0000", 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7960000000000001e-06 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516938+0000", 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0123999999999999e-05 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.516950+0000", 2024-04-24T00:33:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1813e-05 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.517075+0000", 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125019 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.517324+0000", 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024866500000000002 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.517771+0000", 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044676200000000002 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:566 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.518085+0000", 2024-04-24T00:33:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.149044337, 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072053099999999995, 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 566 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518085+0000", 2024-04-24T00:33:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518085+0000", 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518087+0000", 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9599999999999999e-06 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518097+0000", 2024-04-24T00:33:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0210000000000001e-05 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518126+0000", 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8839999999999998e-05 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518141+0000", 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4809e-05 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518262+0000", 2024-04-24T00:33:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012119 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518489+0000", 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022736800000000001 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.518805+0000", 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031615400000000002 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:567 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.get_size in=11b,call rbd.get_features in=15b,call rbd.get_flags in=12b,call,call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.519340+0000", 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14778943999999999, 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014331560000000001, 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 567 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.519340+0000", 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.519340+0000", 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.519342+0000", 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:33:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.519354+0000", 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.519384+0000", 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0020000000000001e-05 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.519402+0000", 2024-04-24T00:33:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8546000000000002e-05 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.519520+0000", 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011733199999999999 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.519756+0000", 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023662499999999999 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.520773+0000", 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001016426 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:568 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call,call rbd.parent_overlap_get in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.521093+0000", 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14603627699999999, 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011797439999999999, 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 568 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.521093+0000", 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.521093+0000", 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.521094+0000", 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.533e-06 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.521102+0000", 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2629999999999997e-06 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.521121+0000", 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9196999999999999e-05 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.521132+0000", 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1066999999999999e-05 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.521281+0000", 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014928900000000001 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.521506+0000", 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022504400000000001 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522273+0000", 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00076635099999999997 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.733 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:569 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.522439+0000", 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14469035699999999, 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00050108900000000003, 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 569 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522439+0000", 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522439+0000", 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522441+0000", 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7460000000000001e-06 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522449+0000", 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1100000000000003e-06 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522469+0000", 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0619999999999999e-05 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522480+0000", 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1027e-05 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522573+0000", 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2929000000000004e-05 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522708+0000", 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013464600000000001 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.522940+0000", 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023201100000000001 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:571 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.524269+0000", 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14286057499999999, 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048984699999999996, 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 571 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.738 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524269+0000", 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524269+0000", 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524270+0000", 2024-04-24T00:33:58.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8870000000000001e-06 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524282+0000", 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524304+0000", 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2316000000000001e-05 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524316+0000", 2024-04-24T00:33:58.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2164000000000001e-05 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524415+0000", 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8572000000000004e-05 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524544+0000", 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012940900000000001 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.524758+0000", 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021428199999999999 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:572 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [watch watch cookie 140203905936496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.525127+0000", 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14200238700000001, 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002099853, 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:58.742 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 572 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525127+0000", 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525127+0000", 2024-04-24T00:33:58.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525128+0000", 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7060000000000001e-06 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525137+0000", 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1070000000000005e-06 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525157+0000", 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0429999999999999e-05 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525169+0000", 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1843e-05 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525282+0000", 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113072 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525403+0000", 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012139 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.525621+0000", 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021739099999999999 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526368+0000", 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00074724700000000004 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526868+0000", 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000500123 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527027+0000", 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158799 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527227+0000", 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019974500000000001 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.747 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:572 3.10 e26/21 ondisk, result = 0)", 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.526687+0000", 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14044251699999999, 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055571600000000004, 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526687+0000", 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526687+0000", 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526688+0000", 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.850000000000001e-07 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526697+0000", 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3349999999999996e-06 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526712+0000", 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5557e-05 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526720+0000", 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0430000000000002e-06 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526824+0000", 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104753 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.526851+0000", 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6222999999999999e-05 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527242+0000", 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039182000000000003 2024-04-24T00:33:58.751 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:573 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.527261+0000", 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.13986843800000001, 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077771800000000005, 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.752 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 573 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527261+0000", 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527261+0000", 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527263+0000", 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527275+0000", 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2619999999999999e-05 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527304+0000", 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8183000000000001e-05 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527319+0000", 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5174000000000001e-05 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527441+0000", 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122199 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.527671+0000", 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023012799999999999 2024-04-24T00:33:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528038+0000", 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036739699999999999 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:574 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.528193+0000", 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.13893640500000001, 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022689860000000002, 2024-04-24T00:33:58.756 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 574 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528193+0000", 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528193+0000", 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528195+0000", 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.827e-06 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528203+0000", 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1200000000000002e-06 2024-04-24T00:33:58.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528223+0000", 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0460000000000001e-05 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528234+0000", 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0987e-05 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528348+0000", 2024-04-24T00:33:58.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011424199999999999 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.528583+0000", 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023503099999999999 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.529113+0000", 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00052979000000000004 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.530148+0000", 2024-04-24T00:33:58.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010345619999999999 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.530203+0000", 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5606000000000003e-05 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.530342+0000", 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013912500000000001 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.530462+0000", 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011923599999999999 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:579 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.534318+0000", 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.13281073900000001, 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00073100800000000001, 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:33:58.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 579 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.534318+0000", 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.534318+0000", 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:58.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.534319+0000", 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8100000000000001e-07 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.534325+0000", 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1569999999999999e-06 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.534337+0000", 2024-04-24T00:33:58.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2700999999999999e-05 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.534343+0000", 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8969999999999996e-06 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.534471+0000", 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127983 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:58.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.534708+0000", 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000237075 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.535049+0000", 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000341214 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:58.766 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:58.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:33:58.768 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:58.760+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:58.868 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:58.864+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:58.937 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:58.937 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:33:58.938 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:33:58.938 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:58.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:33:58.969 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:58.968+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:59.070 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:59.070 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:33:59.070 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:33:59.070 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:33:59.070 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:59.070 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:59.064+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:59.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:33:59.171 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:59.164+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:59.256 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.323902527000001, 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:33:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.165365041000001, 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011455702, 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 37 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084944+0000", 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1999999999999999e-07 2024-04-24T00:33:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084950+0000", 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6200000000000001e-06 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084964+0000", 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3186999999999999e-05 2024-04-24T00:33:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084971+0000", 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3289999999999996e-06 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094143+0000", 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091718640000000001 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094289+0000", 2024-04-24T00:33:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014654499999999999 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094480+0000", 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019091799999999999 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.095845+0000", 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00136509 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096272+0000", 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042710300000000002 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096345+0000", 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2243000000000006e-05 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096399+0000", 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4082999999999997e-05 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.870583342, 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013518105000000001, 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 110 2024-04-24T00:33:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379726+0000", 2024-04-24T00:33:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5770000000000001e-06 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379738+0000", 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1552999999999999e-05 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379758+0000", 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9636e-05 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379773+0000", 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4970000000000001e-05 2024-04-24T00:33:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388655+0000", 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088828440000000009 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388781+0000", 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012547200000000001 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388993+0000", 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021258200000000001 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.392907+0000", 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039138669999999997 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393039+0000", 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013181900000000001 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393112+0000", 2024-04-24T00:33:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2853000000000002e-05 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393243+0000", 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130932 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.870512343000001, 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013665012000000001, 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 111 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379797+0000", 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.173e-06 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379806+0000", 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3370000000000006e-06 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379825+0000", 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8519999999999999e-05 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379839+0000", 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.401e-05 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390285+0000", 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010446162 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390412+0000", 2024-04-24T00:33:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012749899999999999 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390614+0000", 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020195799999999999 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393015+0000", 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024003789999999998 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393268+0000", 2024-04-24T00:33:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025304800000000002 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393356+0000", 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8613e-05 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393461+0000", 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104313 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.870441837000001, 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019315533999999999, 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 112 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379868+0000", 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.274e-06 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380096+0000", 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022802799999999999 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380146+0000", 2024-04-24T00:33:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0358999999999997e-05 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380164+0000", 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7713000000000001e-05 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391250+0000", 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011085928 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391383+0000", 2024-04-24T00:33:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013310899999999999 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391576+0000", 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019287899999999999 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395703+0000", 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041278740000000001 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398947+0000", 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032432149999999998 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399035+0000", 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8635999999999996e-05 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399182+0000", 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014651900000000001 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.870096386, 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019210844000000001, 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 113 2024-04-24T00:33:59.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380213+0000", 2024-04-24T00:33:59.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5829999999999999e-06 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380225+0000", 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1603999999999999e-05 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380244+0000", 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9170000000000001e-05 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380258+0000", 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4382999999999999e-05 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393574+0000", 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013315726999999999 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393703+0000", 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012839200000000001 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393904+0000", 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000201805 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398808+0000", 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049032820000000001 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399200+0000", 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039261 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399310+0000", 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399423+0000", 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011273900000000001 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.87001995, 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019317091000000002, 2024-04-24T00:33:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 114 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380289+0000", 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.173e-06 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380300+0000", 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-05 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380324+0000", 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3047000000000001e-05 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380338+0000", 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4545999999999999e-05 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394634+0000", 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014296233 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394760+0000", 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012605900000000001 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394953+0000", 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019237900000000001 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398924+0000", 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039709589999999996 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399439+0000", 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051536599999999996 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399507+0000", 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8065999999999995e-05 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399605+0000", 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8133000000000004e-05 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.869906488000002, 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020360119999999999, 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 115 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:33:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380404+0000", 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380546+0000", 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014221200000000001 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380561+0000", 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5696e-05 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380572+0000", 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0353999999999999e-05 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395831+0000", 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015259435 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395982+0000", 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150736 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.396234+0000", 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000252334 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399712+0000", 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034776529999999998 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400554+0000", 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00084231699999999998 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400634+0000", 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9270000000000005e-05 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400762+0000", 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012814900000000001 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.869716652000001, 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020559280999999999, 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 116 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380593+0000", 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:33:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380600+0000", 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0160000000000003e-06 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380614+0000", 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4317e-05 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380622+0000", 2024-04-24T00:33:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.397613+0000", 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016991532 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.397762+0000", 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149228 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398030+0000", 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026737499999999998 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400774+0000", 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027444959999999999 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400968+0000", 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193358 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401045+0000", 2024-04-24T00:33:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7440000000000004e-05 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401151+0000", 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105719 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.869673793, 2024-04-24T00:33:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024520601, 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 117 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380635+0000", 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380642+0000", 2024-04-24T00:33:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7700000000000004e-06 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380654+0000", 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1686e-05 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380666+0000", 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2150000000000001e-05 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399787+0000", 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019121461999999999 2024-04-24T00:33:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399921+0000", 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134043 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400127+0000", 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020607500000000001 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401186+0000", 2024-04-24T00:33:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001058738 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.404952+0000", 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037661779999999998 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405042+0000", 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9889999999999995e-05 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405155+0000", 2024-04-24T00:33:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112812 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.360979350000001, 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013799176999999999, 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 133 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889330+0000", 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.339e-06 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889338+0000", 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9610000000000005e-06 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889352+0000", 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:33:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889361+0000", 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8100000000000004e-06 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891442+0000", 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020808860000000001 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891501+0000", 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9314999999999997e-05 2024-04-24T00:33:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891579+0000", 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7163000000000006e-05 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895799+0000", 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042201670000000004 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903071+0000", 2024-04-24T00:33:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072725380000000003 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903101+0000", 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9828e-05 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903128+0000", 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6747e-05 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.360920175, 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014011816999999999, 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 134 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889389+0000", 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3990000000000001e-06 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889469+0000", 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9915999999999999e-05 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889482+0000", 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3215e-05 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889491+0000", 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8179999999999993e-06 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891840+0000", 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023490400000000002 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891898+0000", 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8031000000000001e-05 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891977+0000", 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8512999999999999e-05 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895803+0000", 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00382593 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903233+0000", 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074298970000000004 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903329+0000", 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6718999999999993e-05 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903400+0000", 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0339000000000005e-05 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.360781061000001, 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013963768999999999, 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889528+0000", 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1100000000000004e-07 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889534+0000", 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3539999999999996e-06 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889545+0000", 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0684e-05 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889553+0000", 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2770000000000005e-06 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892602+0000", 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030488920000000001 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892651+0000", 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8905e-05 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892729+0000", 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896463+0000", 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037338810000000001 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903439+0000", 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069760079999999997 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903468+0000", 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9349e-05 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903491+0000", 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2555999999999999e-05 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.36073524, 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01413908, 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889574+0000", 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1399999999999996e-07 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889579+0000", 2024-04-24T00:33:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7810000000000002e-06 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889589+0000", 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3889999999999994e-06 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889597+0000", 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7950000000000008e-06 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893012+0000", 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003415408 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893065+0000", 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2932000000000001e-05 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893157+0000", 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2213999999999995e-05 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896469+0000", 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033114849999999999 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903563+0000", 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070944959999999996 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903649+0000", 2024-04-24T00:33:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5653000000000001e-05 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903712+0000", 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3113000000000004e-05 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.360676566999999, 2024-04-24T00:33:59.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014166239000000001, 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 137 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:33:59.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889713+0000", 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0840999999999994e-05 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889722+0000", 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7329999999999992e-06 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889728+0000", 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8189999999999997e-06 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893442+0000", 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037142540000000002 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893489+0000", 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7271000000000002e-05 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893566+0000", 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6858000000000002e-05 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896481+0000", 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002914551 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903750+0000", 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072693860000000001 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903776+0000", 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6146999999999999e-05 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903798+0000", 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1478999999999999e-05 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:33:59.346 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.360566527, 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015393184000000001, 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 138 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1200000000000003e-07 2024-04-24T00:33:59.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889745+0000", 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3759999999999999e-06 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889751+0000", 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2070000000000003e-06 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889755+0000", 2024-04-24T00:33:59.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0230000000000003e-06 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894238+0000", 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044834610000000002 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894288+0000", 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9434000000000001e-05 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894364+0000", 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5937000000000002e-05 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896492+0000", 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021286870000000002 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905091+0000", 2024-04-24T00:33:59.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085987419999999995 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905116+0000", 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4752e-05 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905135+0000", 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9052999999999999e-05 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.360540875000002, 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015493453000000001, 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 139 2024-04-24T00:33:59.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889768+0000", 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889774+0000", 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0050000000000001e-06 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889781+0000", 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7090000000000002e-06 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889785+0000", 2024-04-24T00:33:59.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8489999999999999e-06 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894637+0000", 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048518320000000004 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894687+0000", 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0102000000000002e-05 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894765+0000", 2024-04-24T00:33:59.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7509000000000006e-05 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896500+0000", 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017353259999999999 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905183+0000", 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086828540000000003 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905208+0000", 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5622999999999998e-05 2024-04-24T00:33:59.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905261+0000", 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2244999999999999e-05 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.360514735999999, 2024-04-24T00:33:59.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018149166000000001, 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:33:59.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889794+0000", 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3200000000000001e-07 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889854+0000", 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0100999999999999e-05 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889861+0000", 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.759e-06 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889865+0000", 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6059999999999996e-06 2024-04-24T00:33:59.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895025+0000", 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051602059999999996 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895081+0000", 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5625999999999999e-05 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895156+0000", 2024-04-24T00:33:59.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4684000000000005e-05 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.897187+0000", 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020309880000000001 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907791+0000", 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010604282 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907877+0000", 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6360000000000007e-05 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907943+0000", 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5222000000000007e-05 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:33:59.364 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.360433280999999, 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01817011, 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 141 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2800000000000002e-07 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889879+0000", 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3359999999999999e-06 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889884+0000", 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.31e-06 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889889+0000", 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8369999999999996e-06 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895829+0000", 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059402509999999997 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895895+0000", 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6210999999999993e-05 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895973+0000", 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7540000000000006e-05 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.897198+0000", 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001225414 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907985+0000", 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010786636 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.908019+0000", 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4258000000000003e-05 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.908045+0000", 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5888999999999999e-05 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 16.566414181999999, 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011747629000000001, 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:33:59.370 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 159 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:33:59.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683895+0000", 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1299999999999998e-07 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683901+0000", 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.823e-06 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683911+0000", 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.004e-05 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683918+0000", 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.294e-06 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.692948+0000", 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090303129999999999 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.693199+0000", 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025027200000000001 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.693589+0000", 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039074299999999998 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695297+0000", 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001707854 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695439+0000", 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014188499999999999 2024-04-24T00:33:59.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695582+0000", 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000142473 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695642+0000", 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0019000000000003e-05 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:33:59.375 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:33:59.377 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:59.372+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:59.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:33:59.451 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T00:33:59.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T00:33:59.479 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:33:59.472+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:33:59.524 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:59.525 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:33:59.525 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:33:59.525 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:33:59.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:33:59.585 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:33:59.580+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:33:59.679 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:33:59.672+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:33:59.705 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:33:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:33:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:33:59.706 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:33:59.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:33:59.780 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:33:59.776+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:33:59.882 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:33:59.876+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:33:59.894 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:33:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:129 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'87, mlcod=16'85)", 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.999428677000001, 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078162660000000005, 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891116+0000", 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.747e-06 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891132+0000", 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5282999999999999e-05 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891161+0000", 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9519e-05 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891175+0000", 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-04-24T00:33:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891315+0000", 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140032 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891354+0000", 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9199999999999997e-05 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898842+0000", 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074878690000000003 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898931+0000", 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8988999999999998e-05 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:130 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'88, mlcod=16'85)", 2024-04-24T00:33:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:33:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.998739628999999, 2024-04-24T00:33:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098350269999999997, 2024-04-24T00:33:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891805+0000", 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.73e-06 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891819+0000", 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.382e-05 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891847+0000", 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8155999999999999e-05 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891867+0000", 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9919999999999999e-05 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892020+0000", 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152139 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892101+0000", 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1738999999999998e-05 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901528+0000", 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094262470000000004 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901639+0000", 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111276 2024-04-24T00:33:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:131 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.998231743000002, 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094484630000000007, 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892313+0000", 2024-04-24T00:33:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7770000000000001e-06 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892385+0000", 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1939000000000004e-05 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892434+0000", 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9023000000000001e-05 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892449+0000", 2024-04-24T00:33:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4263e-05 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892995+0000", 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054614900000000001 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893032+0000", 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6863999999999998e-05 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901646+0000", 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086146460000000001 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901760+0000", 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011380200000000001 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:132 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.997406392999999, 2024-04-24T00:33:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086804629999999994, 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893139+0000", 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.747e-06 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893152+0000", 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.347e-05 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893182+0000", 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9366e-05 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893198+0000", 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6677e-05 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894622+0000", 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014240489999999999 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894661+0000", 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9019e-05 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901769+0000", 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071080149999999996 2024-04-24T00:33:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901817+0000", 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.812e-05 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:133 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'85)", 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.997067082000001, 2024-04-24T00:33:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01116346, 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893478+0000", 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3659999999999999e-06 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893490+0000", 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2784e-05 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893519+0000", 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8983e-05 2024-04-24T00:33:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893544+0000", 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4813000000000001e-05 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895382+0000", 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018376759999999999 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895433+0000", 2024-04-24T00:33:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1093e-05 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904576+0000", 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091431289999999998 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904640+0000", 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3615999999999995e-05 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:134 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'85)", 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.996647012, 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010781628999999999, 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893898+0000", 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4670000000000001e-06 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893990+0000", 2024-04-24T00:33:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2221999999999999e-05 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894021+0000", 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0710000000000002e-05 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894040+0000", 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9303000000000001e-05 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896087+0000", 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020471909999999999 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896120+0000", 2024-04-24T00:33:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3056999999999999e-05 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904643+0000", 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085222530000000005 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904678+0000", 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5426000000000003e-05 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:135 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'85)", 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.995844644999998, 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010018802, 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894701+0000", 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.79e-06 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894713+0000", 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2737e-05 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894743+0000", 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0086e-05 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894764+0000", 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1063000000000001e-05 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897677+0000", 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029129609999999999 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897711+0000", 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3763000000000002e-05 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904682+0000", 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069707090000000003 2024-04-24T00:33:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904718+0000", 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5692999999999999e-05 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:136 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'85)", 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.995463280999999, 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096742749999999995, 2024-04-24T00:33:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895082+0000", 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.64e-06 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895093+0000", 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895122+0000", 2024-04-24T00:33:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8989999999999999e-05 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895162+0000", 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9573000000000002e-05 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898343+0000", 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031815279999999999 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898376+0000", 2024-04-24T00:33:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2610000000000001e-05 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904722+0000", 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006346338 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904754+0000", 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2033000000000003e-05 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:137 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'85)", 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.995028570999999, 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092832910000000008, 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895517+0000", 2024-04-24T00:33:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895604+0000", 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6876e-05 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895632+0000", 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.834e-05 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895649+0000", 2024-04-24T00:33:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7266e-05 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.899032+0000", 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033826339999999998 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.899066+0000", 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3603000000000003e-05 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904756+0000", 2024-04-24T00:33:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056903190000000001 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904798+0000", 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2132999999999998e-05 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:138 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'96, mlcod=16'85)", 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.994238510999999, 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010420479, 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896307+0000", 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7099999999999999e-06 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896321+0000", 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4847e-05 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896348+0000", 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6216000000000001e-05 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896367+0000", 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.965e-05 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.900665+0000", 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042981570000000004 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.900727+0000", 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1682999999999999e-05 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906588+0000", 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058611999999999996 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906725+0000", 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137016 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:139 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'97, mlcod=16'85)", 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:33:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.993894430000001, 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010172130999999999, 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896650+0000", 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.457e-06 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896665+0000", 2024-04-24T00:33:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5085999999999999e-05 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896696+0000", 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0505999999999999e-05 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896713+0000", 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7139999999999999e-05 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901935+0000", 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052216169999999996 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901974+0000", 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9632000000000003e-05 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906732+0000", 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047579370000000003 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906821+0000", 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8756000000000002e-05 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:140 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'85)", 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.993395163999999, 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012183782000000001, 2024-04-24T00:33:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897149+0000", 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.147e-06 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897223+0000", 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4116000000000007e-05 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897251+0000", 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7606e-05 2024-04-24T00:33:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897269+0000", 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7476999999999999e-05 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902553+0000", 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052847359999999999 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902586+0000", 2024-04-24T00:33:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2802000000000002e-05 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909235+0000", 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066493519999999999 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909332+0000", 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6545999999999993e-05 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'85)", 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.992585053999999, 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011427935, 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897960+0000", 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1500000000000002e-06 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897974+0000", 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3890000000000001e-05 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898000+0000", 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.599e-05 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898018+0000", 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7903000000000001e-05 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903913+0000", 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058942839999999996 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903976+0000", 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3847e-05 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909338+0000", 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053611850000000001 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909386+0000", 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8686000000000001e-05 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:33:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8581253850000001, 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097809650000000008, 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 11 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032421+0000", 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3730000000000001e-06 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032435+0000", 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3468999999999999e-05 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032461+0000", 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032473+0000", 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1613999999999999e-05 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032624+0000", 2024-04-24T00:33:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015161999999999999 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032631+0000", 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8700000000000003e-06 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037449+0000", 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048180180000000003 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037551+0000", 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010177 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037724+0000", 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172943 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039447+0000", 2024-04-24T00:33:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017229560000000001 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042096+0000", 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002649069 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042164+0000", 2024-04-24T00:33:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7931999999999999e-05 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042199+0000", 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123000000000002e-05 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:770 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.480364+0000", 2024-04-24T00:33:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.41017899000000002, 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048361020000000001, 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 770 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480364+0000", 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480364+0000", 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480366+0000", 2024-04-24T00:33:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.372e-06 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480375+0000", 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6169999999999996e-06 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480394+0000", 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8281e-05 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480404+0000", 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0304e-05 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480478+0000", 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3910000000000002e-05 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480611+0000", 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013329999999999999 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481235+0000", 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062349200000000001 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.484209+0000", 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002973874 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485098+0000", 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088930300000000003 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485144+0000", 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5666e-05 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485200+0000", 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6983e-05 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:771 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.480762+0000", 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.409781006, 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055778920000000001, 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 771 2024-04-24T00:33:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480762+0000", 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480762+0000", 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480764+0000", 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-06 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480771+0000", 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7700000000000004e-06 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480787+0000", 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6739e-05 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480797+0000", 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7559999999999994e-06 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481743+0000", 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094565100000000004 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481804+0000", 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1775999999999999e-05 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.482027+0000", 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022297199999999999 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485225+0000", 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031973269999999998 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486217+0000", 2024-04-24T00:33:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099223400000000004 2024-04-24T00:33:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486286+0000", 2024-04-24T00:33:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8526000000000001e-05 2024-04-24T00:33:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486340+0000", 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4725999999999997e-05 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:772 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.481122+0000", 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.40942183700000001, 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053661680000000002, 2024-04-24T00:33:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 772 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481122+0000", 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481122+0000", 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481123+0000", 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3999999999999999e-06 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481130+0000", 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6800000000000004e-06 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481145+0000", 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5630000000000001e-05 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481155+0000", 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0051000000000001e-05 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.482486+0000", 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001330323 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.482565+0000", 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8925999999999996e-05 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.482973+0000", 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040814099999999999 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485238+0000", 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022650420000000001 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486389+0000", 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001151074 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486444+0000", 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4913000000000001e-05 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486488+0000", 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3988e-05 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:773 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.481193+0000", 2024-04-24T00:33:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.40935018299999998, 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074095879999999999, 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 773 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481193+0000", 2024-04-24T00:33:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481193+0000", 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481194+0000", 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481200+0000", 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5570000000000003e-06 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481212+0000", 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1911e-05 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481222+0000", 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0662000000000001e-05 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483417+0000", 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002195135 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483490+0000", 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2521000000000006e-05 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:33:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483807+0000", 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031740300000000001 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486550+0000", 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027422280000000002 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.488483+0000", 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001932991 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.488543+0000", 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0028999999999997e-05 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.488603+0000", 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0251000000000003e-05 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:774 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.481503+0000", 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.40904048700000001, 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096041609999999999, 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 774 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481503+0000", 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481503+0000", 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481504+0000", 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3340000000000001e-06 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481510+0000", 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2659999999999998e-06 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481525+0000", 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4858000000000001e-05 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481535+0000", 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2590000000000006e-06 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.484259+0000", 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027241840000000002 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.484330+0000", 2024-04-24T00:33:59.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1196000000000001e-05 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.484632+0000", 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030203599999999999 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486575+0000", 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019433569999999999 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491056+0000", 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004480436 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491084+0000", 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8265000000000001e-05 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491107+0000", 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2969999999999999e-05 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:775 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.483006+0000", 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.40753709300000002, 2024-04-24T00:33:59.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084191790000000006, 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 775 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:33:59.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483006+0000", 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483006+0000", 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483007+0000", 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5099999999999998e-07 2024-04-24T00:33:59.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483012+0000", 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1100000000000002e-06 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483023+0000", 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0807e-05 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483046+0000", 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2974000000000001e-05 2024-04-24T00:33:59.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485274+0000", 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002227499 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485369+0000", 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5424e-05 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:33:59.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485757+0000", 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038762099999999999 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486926+0000", 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011698780000000001 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491211+0000", 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042842310000000003 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491311+0000", 2024-04-24T00:33:59.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491425+0000", 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114499 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.000 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:00.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:00.002 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:33:59.996+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:00.103 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:00.096+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:00.174 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:00.175 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:00.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:00.233 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:00.228+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:00.304 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:00.296+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:00.361 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:00.361 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:00.362 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:00.362 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:00.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:00.417 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:00.412+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T00:34:00.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:00.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:34:00.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T00:34:00.542 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:00.536+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:00.569 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:00.569 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:7 3.1a e23/21 3:5cea7035:::rbd_info:head v 23'1, mlcod=0'0)", 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:54.023528+0000", 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.5433976219999996, 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001701541, 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.023528+0000", 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.023528+0000", 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.023529+0000", 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4000000000000002e-07 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.023538+0000", 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7139999999999999e-06 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.023564+0000", 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6942999999999999e-05 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.023592+0000", 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7707000000000001e-05 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.023707+0000", 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115089 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.023757+0000", 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9999e-05 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.025135+0000", 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013778799999999999 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:54.025230+0000", 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4668999999999998e-05 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:9 3.1a e24/21 3:5cea7035:::rbd_info:head v 24'2, mlcod=0'0)", 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.024296+0000", 2024-04-24T00:34:00.574 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.542630323, 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005262241, 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.024296+0000", 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.024296+0000", 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.024296+0000", 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.84e-07 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.024318+0000", 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2195e-05 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.024327+0000", 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8850000000000005e-06 2024-04-24T00:34:00.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.024339+0000", 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1834000000000001e-05 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.024403+0000", 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3825000000000006e-05 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.024420+0000", 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7282000000000002e-05 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.029502+0000", 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050824700000000004 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.029558+0000", 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5466000000000001e-05 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10 3.1c e24/21 3:3831950c:::rbd_directory:head v 24'1, mlcod=0'0)", 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.030672+0000", 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.5362535629999998, 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011407380000000001, 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.030672+0000", 2024-04-24T00:34:00.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.030672+0000", 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.030674+0000", 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.375e-06 2024-04-24T00:34:00.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.030685+0000", 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1523e-05 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.030699+0000", 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3923e-05 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.030708+0000", 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8170000000000005e-06 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.030799+0000", 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0600999999999995e-05 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.030839+0000", 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0720000000000003e-05 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.031773+0000", 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093357300000000004 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.031813+0000", 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0206000000000001e-05 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:12 3.16 e24/21 3:6cd04fb2:::rbd_header.115e92aeb10c:head v 24'1, mlcod=0'0)", 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.048203+0000", 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.5187228729999998, 2024-04-24T00:34:00.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011573499999999999, 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.048203+0000", 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.048203+0000", 2024-04-24T00:34:00.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.048204+0000", 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0900000000000001e-07 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.048217+0000", 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3389e-05 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.048226+0000", 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2739999999999996e-06 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.048237+0000", 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1e-05 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.048314+0000", 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6225999999999998e-05 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.048336+0000", 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1943999999999999e-05 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.049328+0000", 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099209600000000004 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.049360+0000", 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2712000000000001e-05 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:25 3.16 e24/21 3:6cd04fb2:::rbd_header.115e92aeb10c:head v 24'2, mlcod=0'0)", 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.064748+0000", 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.5021775069999999, 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090189700000000003, 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.064748+0000", 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.064748+0000", 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.064750+0000", 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.843e-06 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.064762+0000", 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1396999999999999e-05 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.064790+0000", 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8042999999999999e-05 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.064803+0000", 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3546e-05 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.064911+0000", 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107649 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.064954+0000", 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3316000000000003e-05 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.065613+0000", 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065932499999999999 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.065650+0000", 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6777999999999999e-05 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:523 3.16 e26/21 3:6cd04fb2:::rbd_header.115e92aeb10c:head v 26'46, mlcod=25'44)", 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:58.448329+0000", 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.1185968430000002, 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085736799999999995, 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.448329+0000", 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.448329+0000", 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.448329+0000", 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3299999999999998e-07 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.448335+0000", 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8189999999999997e-06 2024-04-24T00:34:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.448369+0000", 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3954000000000001e-05 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.448389+0000", 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9391000000000001e-05 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.448464+0000", 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5013999999999997e-05 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.448484+0000", 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0681000000000001e-05 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.449148+0000", 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066361600000000001 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.449186+0000", 2024-04-24T00:34:00.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8559999999999997e-05 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:525 3.1a e26/21 3:5b11cb80:::rbd_object_map.115e92aeb10c.0000000000000003:head v 26'8, mlcod=25'6)", 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:58.452850+0000", 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.114076302, 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010678790000000001, 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.452850+0000", 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.452850+0000", 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.452850+0000", 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000002e-07 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.452855+0000", 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5300000000000004e-06 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.452880+0000", 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4312999999999999e-05 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.452895+0000", 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4949999999999999e-05 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.452981+0000", 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6341000000000006e-05 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.452999+0000", 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8434e-05 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.453859+0000", 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085954300000000005 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.453917+0000", 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8480999999999998e-05 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:537 3.16 e26/21 3:6cd04fb2:::rbd_header.115e92aeb10c:head v 26'47, mlcod=25'45)", 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:58.479136+0000", 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0877899360000001, 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078809699999999997, 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479136+0000", 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479136+0000", 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479138+0000", 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9199999999999998e-06 2024-04-24T00:34:00.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479150+0000", 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2417e-05 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479172+0000", 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1861e-05 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479185+0000", 2024-04-24T00:34:00.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3334e-05 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479260+0000", 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4919000000000006e-05 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479280+0000", 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9633999999999999e-05 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479886+0000", 2024-04-24T00:34:00.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060575199999999998 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.479924+0000", 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8260000000000003e-05 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:547 3.16 e26/21 3:6cd04fb2:::rbd_header.115e92aeb10c:head v 26'48, mlcod=26'46)", 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:58.493064+0000", 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0738621519999998, 2024-04-24T00:34:00.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063117499999999999, 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493064+0000", 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493064+0000", 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493065+0000", 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493072+0000", 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2139999999999999e-06 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493085+0000", 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3247999999999999e-05 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493095+0000", 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4020000000000008e-06 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493152+0000", 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6813e-05 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493165+0000", 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.344e-05 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493668+0000", 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050288000000000004 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.493695+0000", 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7025e-05 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:555 3.1c e26/21 3:3831950c:::rbd_directory:head v 26'4, mlcod=24'2)", 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:58.500895+0000", 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0660308970000001, 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084220900000000001, 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.500895+0000", 2024-04-24T00:34:00.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.500895+0000", 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.500897+0000", 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.756e-06 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.500908+0000", 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1240000000000001e-05 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.500935+0000", 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6627000000000001e-05 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.500953+0000", 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8403e-05 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.501071+0000", 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118458 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.501085+0000", 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3331e-05 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.501705+0000", 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000620303 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:58.501737+0000", 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2091000000000002e-05 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:607 3.19 3:9d961f42:::rbd_data.115e92aeb10c.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.469664+0000", 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0972617220000001, 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026326000000000002, 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 607 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.469664+0000", 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.469664+0000", 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.469667+0000", 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5610000000000001e-06 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.469676+0000", 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7499999999999998e-06 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.469705+0000", 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8776999999999999e-05 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.469717+0000", 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1610999999999999e-05 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.469802+0000", 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4708000000000003e-05 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.469927+0000", 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012585299999999999 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:623 3.19 3:9d961f42:::rbd_data.115e92aeb10c.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.470187+0000", 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.096738845, 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035667499999999998, 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 623 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470187+0000", 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470187+0000", 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470188+0000", 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.189e-06 2024-04-24T00:34:00.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470194+0000", 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6119999999999998e-06 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470211+0000", 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7345000000000001e-05 2024-04-24T00:34:00.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470220+0000", 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6729999999999998e-06 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470444+0000", 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022430400000000001 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470544+0000", 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9551999999999998e-05 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:639 3.19 3:9d961f42:::rbd_data.115e92aeb10c.0000000000000006:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.470674+0000", 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0962514539999999, 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021317499999999999, 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.623 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 639 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470674+0000", 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470674+0000", 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470676+0000", 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.64e-06 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470686+0000", 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0288e-05 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470712+0000", 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5279e-05 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470724+0000", 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2568999999999999e-05 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470793+0000", 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8485999999999994e-05 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470888+0000", 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4913000000000004e-05 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.627 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:649 3.19 3:9e32dd40:::rbd_data.115e92aeb10c.0000000000000018:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.470899+0000", 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.096027332, 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020884899999999999, 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 649 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470899+0000", 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470899+0000", 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470900+0000", 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3629999999999999e-06 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470907+0000", 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000003e-06 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470922+0000", 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4771e-05 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.470933+0000", 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1107999999999999e-05 2024-04-24T00:34:00.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471007+0000", 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4008e-05 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471107+0000", 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010069 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:656 3.4 3:203ce3e1:::rbd_data.115e92aeb10c.0000000000000023:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.471102+0000", 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0958243649999999, 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022374999999999999, 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 656 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471102+0000", 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471102+0000", 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471103+0000", 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.246e-06 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471109+0000", 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0639999999999999e-06 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471124+0000", 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471133+0000", 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3340000000000008e-06 2024-04-24T00:34:00.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471212+0000", 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.907e-05 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471325+0000", 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011291299999999999 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:672 3.4 3:203ce3e1:::rbd_data.115e92aeb10c.0000000000000023:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.471587+0000", 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0953386810000001, 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195183, 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 672 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471587+0000", 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471587+0000", 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471588+0000", 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0440000000000001e-06 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471594+0000", 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3680000000000001e-06 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471613+0000", 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.914e-05 2024-04-24T00:34:00.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471622+0000", 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8219999999999996e-06 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471684+0000", 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2131999999999994e-05 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.471782+0000", 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8677000000000004e-05 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:688 3.4 3:203ce3e1:::rbd_data.115e92aeb10c.0000000000000023:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.473407+0000", 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0935190610000001, 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0002429, 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.640 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 688 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473407+0000", 2024-04-24T00:34:00.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473407+0000", 2024-04-24T00:34:00.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473409+0000", 2024-04-24T00:34:00.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1059999999999998e-06 2024-04-24T00:34:00.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473416+0000", 2024-04-24T00:34:00.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1740000000000003e-06 2024-04-24T00:34:00.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473439+0000", 2024-04-24T00:34:00.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2628000000000001e-05 2024-04-24T00:34:00.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473452+0000", 2024-04-24T00:34:00.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2846e-05 2024-04-24T00:34:00.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473523+0000", 2024-04-24T00:34:00.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1895000000000002e-05 2024-04-24T00:34:00.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473650+0000", 2024-04-24T00:34:00.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012625099999999999 2024-04-24T00:34:00.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:704 3.4 3:203ce3e1:::rbd_data.115e92aeb10c.0000000000000023:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.473690+0000", 2024-04-24T00:34:00.649 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0932360999999999, 2024-04-24T00:34:00.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078556100000000005, 2024-04-24T00:34:00.649 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.649 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.650 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 704 2024-04-24T00:34:00.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.650 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473690+0000", 2024-04-24T00:34:00.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473690+0000", 2024-04-24T00:34:00.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473691+0000", 2024-04-24T00:34:00.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.342e-06 2024-04-24T00:34:00.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473697+0000", 2024-04-24T00:34:00.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3419999999999996e-06 2024-04-24T00:34:00.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473711+0000", 2024-04-24T00:34:00.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3312999999999999e-05 2024-04-24T00:34:00.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.473720+0000", 2024-04-24T00:34:00.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5480000000000007e-06 2024-04-24T00:34:00.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.474348+0000", 2024-04-24T00:34:00.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000628091 2024-04-24T00:34:00.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.474475+0000", 2024-04-24T00:34:00.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012692500000000001 2024-04-24T00:34:00.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.658 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:804 2.2 2:535385f0:::rbd_children:head [call rbd.get_children in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:00.658 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.528836+0000", 2024-04-24T00:34:00.658 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.038089949, 2024-04-24T00:34:00.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021575899999999999, 2024-04-24T00:34:00.658 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:00.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:00.659 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:00.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:00.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:00.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:00.659 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 804 2024-04-24T00:34:00.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.660 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:00.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:00.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.528836+0000", 2024-04-24T00:34:00.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:00.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.528836+0000", 2024-04-24T00:34:00.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:00.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:00.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.528838+0000", 2024-04-24T00:34:00.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.491e-06 2024-04-24T00:34:00.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:00.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.528849+0000", 2024-04-24T00:34:00.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0696000000000001e-05 2024-04-24T00:34:00.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:00.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.528873+0000", 2024-04-24T00:34:00.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:34:00.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:00.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.528883+0000", 2024-04-24T00:34:00.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0454e-05 2024-04-24T00:34:00.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:00.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.528951+0000", 2024-04-24T00:34:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7455999999999999e-05 2024-04-24T00:34:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.529052+0000", 2024-04-24T00:34:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010078599999999999 2024-04-24T00:34:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:00.666 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:00.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:00.668 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:00.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:00.669 INFO:teuthology.orchestra.run.smithi064.stdout:174 punch from 0x208b924 to 0x2090aef, (0x51cb bytes) 2024-04-24T00:34:00.669 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:00.670 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:00.670 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:00.670 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:00.671 INFO:teuthology.orchestra.run.smithi064.stdout:175 write 0xe9ef4e7 thru 0xe9fd959 (0xe473 bytes) 2024-04-24T00:34:00.671 INFO:teuthology.orchestra.run.smithi064.stdout:176 read 0x500026 thru 0x5083e0 (0x83bb bytes) 2024-04-24T00:34:00.671 INFO:teuthology.orchestra.run.smithi064.stdout:177 write 0x2755759 thru 0x275cd03 (0x75ab bytes) 2024-04-24T00:34:00.671 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:00.664+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:00.676 INFO:teuthology.orchestra.run.smithi064.stdout:179 punch from 0x57d6fc4 to 0x57e14fc, (0xa538 bytes) 2024-04-24T00:34:00.683 INFO:teuthology.orchestra.run.smithi064.stdout:180 trunc from 0xe9fd95a to 0x9bffe95 2024-04-24T00:34:00.694 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:00.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:00.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:00.730 INFO:teuthology.orchestra.run.smithi064.stdout:181 write 0xcaa144e thru 0xcaab07c (0x9c2f bytes) 2024-04-24T00:34:00.737 INFO:teuthology.orchestra.run.smithi064.stdout:182 write 0x1a6b6f8 thru 0x1a6d3ee (0x1cf7 bytes) 2024-04-24T00:34:00.746 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:00.746 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:00.758 INFO:teuthology.orchestra.run.smithi064.stdout:185 write 0xde87aeb thru 0xde89f0d (0x2423 bytes) 2024-04-24T00:34:00.761 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:00.761 INFO:teuthology.orchestra.run.smithi064.stdout:187 read 0x2c45cbc thru 0x2c52b35 (0xce7a bytes) 2024-04-24T00:34:00.761 INFO:teuthology.orchestra.run.smithi064.stdout:188 write 0x20872e7 thru 0x208ab78 (0x3892 bytes) 2024-04-24T00:34:00.765 INFO:teuthology.orchestra.run.smithi064.stdout:189 read 0x3ab92bf thru 0x3ac5ae7 (0xc829 bytes) 2024-04-24T00:34:00.765 INFO:teuthology.orchestra.run.smithi064.stdout:192 punch from 0xb0a081a to 0xb0abfbf, (0xb7a5 bytes) 2024-04-24T00:34:00.766 INFO:teuthology.orchestra.run.smithi064.stdout:193 punch from 0x979892 to 0x983bf5, (0xa363 bytes) 2024-04-24T00:34:00.766 INFO:teuthology.orchestra.run.smithi064.stdout:194 writesame 0x9d94b4d thru 0x9d974de data_size 0x11(0x2992 bytes) 2024-04-24T00:34:00.769 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:00.764+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:00.770 INFO:teuthology.orchestra.run.smithi064.stdout:195 read 0xd02e5c4 thru 0xd0374e0 (0x8f1d bytes) 2024-04-24T00:34:00.771 INFO:teuthology.orchestra.run.smithi064.stdout:196 punch from 0x4cb02f3 to 0x4cba595, (0xa2a2 bytes) 2024-04-24T00:34:00.771 INFO:teuthology.orchestra.run.smithi064.stdout:197 write 0xb34c898 thru 0xb350fbd (0x4726 bytes) 2024-04-24T00:34:00.775 INFO:teuthology.orchestra.run.smithi064.stdout:198 read 0xa61c2d5 thru 0xa62b71d (0xf449 bytes) 2024-04-24T00:34:00.775 INFO:teuthology.orchestra.run.smithi064.stdout:199 read 0xb24abf0 thru 0xb25289f (0x7cb0 bytes) 2024-04-24T00:34:00.776 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:00.776 INFO:teuthology.orchestra.run.smithi064.stdout:202 write 0xb960924 thru 0xb965336 (0x4a13 bytes) 2024-04-24T00:34:00.781 INFO:teuthology.orchestra.run.smithi064.stdout:203 writesame 0x65416e1 thru 0x6547ddc data_size 0x34(0x66fc bytes) 2024-04-24T00:34:00.784 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:00.784 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:00.787 INFO:teuthology.orchestra.run.smithi064.stdout:205 read 0x67cb0f4 thru 0x67d67c3 (0xb6d0 bytes) 2024-04-24T00:34:00.787 INFO:teuthology.orchestra.run.smithi064.stdout:206 read 0x19b46cf thru 0x19b8009 (0x393b bytes) 2024-04-24T00:34:00.788 INFO:teuthology.orchestra.run.smithi064.stdout:207 read 0xd0d81c7 thru 0xd0e5409 (0xd243 bytes) 2024-04-24T00:34:00.788 INFO:teuthology.orchestra.run.smithi064.stdout:209 read 0x3bcf5c6 thru 0x3bd1822 (0x225d bytes) 2024-04-24T00:34:00.789 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:00.790 INFO:teuthology.orchestra.run.smithi064.stdout:212 trunc from 0xde89f0e to 0xd45cdba 2024-04-24T00:34:00.805 INFO:teuthology.orchestra.run.smithi064.stdout:213 read 0x5ebf224 thru 0x5ec6dbd (0x7b9a bytes) 2024-04-24T00:34:00.805 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:00.805 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:00.806 INFO:teuthology.orchestra.run.smithi064.stdout:214 punch from 0xbd5009e to 0xbd52fed, (0x2f4f bytes) 2024-04-24T00:34:00.806 INFO:teuthology.orchestra.run.smithi064.stdout:215 write 0xba57b5e thru 0xba65a4e (0xdef1 bytes) 2024-04-24T00:34:00.807 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:00.808 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:00.816 INFO:teuthology.orchestra.run.smithi064.stdout:216 read 0x6a07e9d thru 0x6a0f050 (0x71b4 bytes) 2024-04-24T00:34:00.818 INFO:teuthology.orchestra.run.smithi064.stdout:217 trunc from 0xd45cdba to 0xb942452 2024-04-24T00:34:00.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:00.835 INFO:teuthology.orchestra.run.smithi064.stdout:218 punch from 0x3c37583 to 0x3c3df9b, (0x6a18 bytes) 2024-04-24T00:34:00.835 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:00.836 INFO:teuthology.orchestra.run.smithi064.stdout:220 write 0x36ea7f4 thru 0x36f6e4c (0xc659 bytes) 2024-04-24T00:34:00.844 INFO:teuthology.orchestra.run.smithi064.stdout:221 trunc from 0xb942452 to 0xb81cc3f 2024-04-24T00:34:00.859 INFO:teuthology.orchestra.run.smithi064.stdout:222 read 0xa77255d thru 0xa78161b (0xf0bf bytes) 2024-04-24T00:34:00.870 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:00.864+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:00.873 INFO:teuthology.orchestra.run.smithi064.stdout:223 trunc from 0xb81cc3f to 0x36db9dd 2024-04-24T00:34:00.903 INFO:teuthology.orchestra.run.smithi064.stdout:224 read 0x3f0adc thru 0x3f5c3b (0x5160 bytes) 2024-04-24T00:34:00.931 INFO:teuthology.orchestra.run.smithi064.stdout:225 write 0xcb0196f thru 0xcb070aa (0x573c bytes) 2024-04-24T00:34:00.938 INFO:teuthology.orchestra.run.smithi064.stdout:226 write 0xaed4c4a thru 0xaed8554 (0x390b bytes) 2024-04-24T00:34:00.958 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:00.958 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:00.959 INFO:teuthology.orchestra.run.smithi064.stdout:227 writesame 0xd4c9ac1 thru 0xd4d5425 data_size 0x1(0xb965 bytes) 2024-04-24T00:34:00.960 INFO:teuthology.orchestra.run.smithi064.stdout:228 write 0xa2688ff thru 0xa26ee68 (0x656a bytes) 2024-04-24T00:34:00.966 INFO:teuthology.orchestra.run.smithi064.stdout:229 punch from 0x528c263 to 0x529a29e, (0xe03b bytes) 2024-04-24T00:34:00.966 INFO:teuthology.orchestra.run.smithi064.stdout:230 writesame 0x39a376d thru 0x39abf83 data_size 0x4f(0x8817 bytes) 2024-04-24T00:34:00.971 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:00.964+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:00.973 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:00.973 INFO:teuthology.orchestra.run.smithi064.stdout:232 write 0x96640b9 thru 0x966549e (0x13e6 bytes) 2024-04-24T00:34:00.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:00.981 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:00.981 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:00.981 INFO:teuthology.orchestra.run.smithi064.stdout:235 writesame 0x7739ee0 thru 0x77467c1 data_size 0x9(0xc8e2 bytes) 2024-04-24T00:34:01.013 INFO:teuthology.orchestra.run.smithi064.stdout:236 trunc from 0xd4d5426 to 0xc1c0fec 2024-04-24T00:34:01.032 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.032 INFO:teuthology.orchestra.run.smithi064.stdout:238 write 0xb0ba0d0 thru 0xb0c758d (0xd4be bytes) 2024-04-24T00:34:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:241 read 0x24dab24 thru 0x24e0c5d (0x613a bytes) 2024-04-24T00:34:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:243 read 0xac09cd4 thru 0xac0ffbc (0x62e9 bytes) 2024-04-24T00:34:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:244 read 0x216decd thru 0x2170e4b (0x2f7f bytes) 2024-04-24T00:34:01.037 INFO:teuthology.orchestra.run.smithi064.stdout:245 writesame 0x893d40c thru 0x894875f data_size 0x2e(0xb354 bytes) 2024-04-24T00:34:01.044 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.044 INFO:teuthology.orchestra.run.smithi064.stdout:248 write 0xbe8a9f1 thru 0xbe99766 (0xed76 bytes) 2024-04-24T00:34:01.048 INFO:teuthology.orchestra.run.smithi064.stdout:249 read 0x60aadb7 thru 0x60b35d3 (0x881d bytes) 2024-04-24T00:34:01.048 INFO:teuthology.orchestra.run.smithi064.stdout:250 read 0x2029c5 thru 0x2127e5 (0xfe21 bytes) 2024-04-24T00:34:01.049 INFO:teuthology.orchestra.run.smithi064.stdout:251 read 0x655eb3f thru 0x656c3ec (0xd8ae bytes) 2024-04-24T00:34:01.049 INFO:teuthology.orchestra.run.smithi064.stdout:252 punch from 0x4c9678d to 0x4ca047a, (0x9ced bytes) 2024-04-24T00:34:01.049 INFO:teuthology.orchestra.run.smithi064.stdout:253 write 0x3df2c3a thru 0x3df783d (0x4c04 bytes) 2024-04-24T00:34:01.053 INFO:teuthology.orchestra.run.smithi064.stdout:255 punch from 0xb8e2c2d to 0xb8e97b1, (0x6b84 bytes) 2024-04-24T00:34:01.053 INFO:teuthology.orchestra.run.smithi064.stdout:258 write 0x35ef6c3 thru 0x35f236f (0x2cad bytes) 2024-04-24T00:34:01.056 INFO:teuthology.orchestra.run.smithi064.stdout:259 write 0x37819bb thru 0x379057f (0xebc5 bytes) 2024-04-24T00:34:01.065 INFO:teuthology.orchestra.run.smithi064.stdout:260 writesame 0x33b63cb thru 0x33c2f8b data_size 0x1(0xcbc1 bytes) 2024-04-24T00:34:01.071 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:01.068+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:01.107 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:01.107 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:01.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:01.167 INFO:teuthology.orchestra.run.smithi064.stdout:261 read 0x9a056b9 thru 0x9a0cd98 (0x76e0 bytes) 2024-04-24T00:34:01.168 INFO:teuthology.orchestra.run.smithi064.stdout:263 writesame 0x6d8a4b9 thru 0x6d90024 data_size 0x2(0x5b6c bytes) 2024-04-24T00:34:01.172 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:01.168+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:01.190 INFO:teuthology.orchestra.run.smithi064.stdout:264 read 0x9e9edc8 thru 0x9ea16ae (0x28e7 bytes) 2024-04-24T00:34:01.191 INFO:teuthology.orchestra.run.smithi064.stdout:265 read 0x4835a55 thru 0x483e453 (0x89ff bytes) 2024-04-24T00:34:01.192 INFO:teuthology.orchestra.run.smithi064.stdout:266 punch from 0x593f159 to 0x5942f98, (0x3e3f bytes) 2024-04-24T00:34:01.192 INFO:teuthology.orchestra.run.smithi064.stdout:267 read 0x7b52331 thru 0x7b5ced9 (0xaba9 bytes) 2024-04-24T00:34:01.193 INFO:teuthology.orchestra.run.smithi064.stdout:268 read 0x9747053 thru 0x974b306 (0x42b4 bytes) 2024-04-24T00:34:01.194 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.194 INFO:teuthology.orchestra.run.smithi064.stdout:270 writesame 0x39fd2e6 thru 0x3a0642c data_size 0x4d(0x9147 bytes) 2024-04-24T00:34:01.196 INFO:teuthology.orchestra.run.smithi064.stdout:271 write 0xb233539 thru 0xb2378ca (0x4392 bytes) 2024-04-24T00:34:01.203 INFO:teuthology.orchestra.run.smithi064.stdout:272 read 0x481d382 thru 0x482888a (0xb509 bytes) 2024-04-24T00:34:01.204 INFO:teuthology.orchestra.run.smithi064.stdout:273 writesame 0x524af6a thru 0x52525e8 data_size 0x1(0x767f bytes) 2024-04-24T00:34:01.259 INFO:teuthology.orchestra.run.smithi064.stdout:274 read 0x21bcb28 thru 0x21c9266 (0xc73f bytes) 2024-04-24T00:34:01.259 INFO:teuthology.orchestra.run.smithi064.stdout:275 read 0x998c63c thru 0x999453c (0x7f01 bytes) 2024-04-24T00:34:01.262 INFO:teuthology.orchestra.run.smithi064.stdout:277 trunc from 0xc1c0fec to 0xb7eb086 2024-04-24T00:34:01.273 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:01.268+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:01.280 INFO:teuthology.orchestra.run.smithi064.stdout:278 write 0x387211c thru 0x387d26c (0xb151 bytes) 2024-04-24T00:34:01.290 INFO:teuthology.orchestra.run.smithi064.stdout:279 punch from 0x9003ad3 to 0x90137f8, (0xfd25 bytes) 2024-04-24T00:34:01.290 INFO:teuthology.orchestra.run.smithi064.stdout:280 write 0xae98101 thru 0xae9f795 (0x7695 bytes) 2024-04-24T00:34:01.304 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:01.304 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:01.310 INFO:teuthology.orchestra.run.smithi064.stdout:281 write 0xcf44a10 thru 0xcf50c43 (0xc234 bytes) 2024-04-24T00:34:01.316 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:01.334 INFO:teuthology.orchestra.run.smithi064.stdout:284 write 0xe79a0bb thru 0xe7a4955 (0xa89b bytes) 2024-04-24T00:34:01.340 INFO:teuthology.orchestra.run.smithi064.stdout:285 write 0x59e6cdc thru 0x59f2f1e (0xc243 bytes) 2024-04-24T00:34:01.358 INFO:teuthology.orchestra.run.smithi064.stdout:287 trunc from 0xe7a4956 to 0x7e7d4c4 2024-04-24T00:34:01.374 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:01.368+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:01.409 INFO:teuthology.orchestra.run.smithi064.stdout:288 write 0xde204f8 thru 0xde276c7 (0x71d0 bytes) 2024-04-24T00:34:01.415 INFO:teuthology.orchestra.run.smithi064.stdout:289 read 0x9b2b40c thru 0x9b328f8 (0x74ed bytes) 2024-04-24T00:34:01.416 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.416 INFO:teuthology.orchestra.run.smithi064.stdout:291 write 0xaabdf69 thru 0xaac3a60 (0x5af8 bytes) 2024-04-24T00:34:01.424 INFO:teuthology.orchestra.run.smithi064.stdout:292 write 0xd61d1f0 thru 0xd6286ae (0xb4bf bytes) 2024-04-24T00:34:01.435 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.435 INFO:teuthology.orchestra.run.smithi064.stdout:294 write 0x5c9d92a thru 0x5ca9c57 (0xc32e bytes) 2024-04-24T00:34:01.443 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.443 INFO:teuthology.orchestra.run.smithi064.stdout:297 write 0x43786c3 thru 0x4378801 (0x13f bytes) 2024-04-24T00:34:01.447 INFO:teuthology.orchestra.run.smithi064.stdout:298 read 0x1cd75b1 thru 0x1ce294f (0xb39f bytes) 2024-04-24T00:34:01.448 INFO:teuthology.orchestra.run.smithi064.stdout:299 read 0xcddbf0f thru 0xcde5452 (0x9544 bytes) 2024-04-24T00:34:01.448 INFO:teuthology.orchestra.run.smithi064.stdout:300 punch from 0x8a2d4f8 to 0x8a3d3dd, (0xfee5 bytes) 2024-04-24T00:34:01.449 INFO:teuthology.orchestra.run.smithi064.stdout:301 punch from 0xb5d152f to 0xb5db32d, (0x9dfe bytes) 2024-04-24T00:34:01.465 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T00:34:01.465 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:34:01.466 INFO:teuthology.orchestra.run.smithi064.stdout:302 trunc from 0xde276c8 to 0x2ac97d3 2024-04-24T00:34:01.469 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:01.469 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:01.474 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:01.468+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:01.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:01.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-24T00:34:01.507 INFO:teuthology.orchestra.run.smithi064.stdout:303 punch from 0x20237ca to 0x2032b5a, (0xf390 bytes) 2024-04-24T00:34:01.507 INFO:teuthology.orchestra.run.smithi064.stdout:304 punch from 0x148d4dd to 0x148e7e7, (0x130a bytes) 2024-04-24T00:34:01.507 INFO:teuthology.orchestra.run.smithi064.stdout:305 write 0x275733c thru 0x27663af (0xf074 bytes) 2024-04-24T00:34:01.517 INFO:teuthology.orchestra.run.smithi064.stdout:306 read 0x225207f thru 0x2257955 (0x58d7 bytes) 2024-04-24T00:34:01.517 INFO:teuthology.orchestra.run.smithi064.stdout:307 read 0x1c98bef thru 0x1ca6e21 (0xe233 bytes) 2024-04-24T00:34:01.544 INFO:teuthology.orchestra.run.smithi064.stdout:308 write 0xea76c51 thru 0xea80059 (0x9409 bytes) 2024-04-24T00:34:01.548 INFO:teuthology.orchestra.run.smithi064.stdout:309 read 0x54a24e5 thru 0x54ab181 (0x8c9d bytes) 2024-04-24T00:34:01.549 INFO:teuthology.orchestra.run.smithi064.stdout:310 write 0xaa9bed thru 0xaabcc5 (0x20d9 bytes) 2024-04-24T00:34:01.551 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.551 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.551 INFO:teuthology.orchestra.run.smithi064.stdout:314 read 0x4da4fdb thru 0x4db4329 (0xf34f bytes) 2024-04-24T00:34:01.552 INFO:teuthology.orchestra.run.smithi064.stdout:315 read 0xc7194c0 thru 0xc721d29 (0x886a bytes) 2024-04-24T00:34:01.552 INFO:teuthology.orchestra.run.smithi064.stdout:316 punch from 0x7457f67 to 0x745b378, (0x3411 bytes) 2024-04-24T00:34:01.552 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.552 INFO:teuthology.orchestra.run.smithi064.stdout:318 read 0x8c9d9d1 thru 0x8ca9047 (0xb677 bytes) 2024-04-24T00:34:01.552 INFO:teuthology.orchestra.run.smithi064.stdout:319 read 0xe19988f thru 0xe1a0c19 (0x738b bytes) 2024-04-24T00:34:01.552 INFO:teuthology.orchestra.run.smithi064.stdout:320 writesame 0x6c4f485 thru 0x6c58ab7 data_size 0x7(0x9633 bytes) 2024-04-24T00:34:01.566 INFO:teuthology.orchestra.run.smithi064.stdout:322 punch from 0xb35f1fd to 0xb35f935, (0x738 bytes) 2024-04-24T00:34:01.566 INFO:teuthology.orchestra.run.smithi064.stdout:323 read 0x78793b6 thru 0x7886538 (0xd183 bytes) 2024-04-24T00:34:01.566 INFO:teuthology.orchestra.run.smithi064.stdout:324 read 0x164fb41 thru 0x1659767 (0x9c27 bytes) 2024-04-24T00:34:01.566 INFO:teuthology.orchestra.run.smithi064.stdout:325 read 0xe35303 thru 0xe445d8 (0xf2d6 bytes) 2024-04-24T00:34:01.568 INFO:teuthology.orchestra.run.smithi064.stdout:326 read 0x2402bcb thru 0x2403900 (0xd36 bytes) 2024-04-24T00:34:01.568 INFO:teuthology.orchestra.run.smithi064.stdout:327 read 0x545f440 thru 0x546a2df (0xaea0 bytes) 2024-04-24T00:34:01.569 INFO:teuthology.orchestra.run.smithi064.stdout:328 writesame 0x2b3954b thru 0x2b3ce7a data_size 0x50(0x3930 bytes) 2024-04-24T00:34:01.571 INFO:teuthology.orchestra.run.smithi064.stdout:329 read 0x9ddc217 thru 0x9de50c4 (0x8eae bytes) 2024-04-24T00:34:01.572 INFO:teuthology.orchestra.run.smithi064.stdout:331 read 0x56a33f2 thru 0x56abde5 (0x89f4 bytes) 2024-04-24T00:34:01.576 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:01.572+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:01.583 INFO:teuthology.orchestra.run.smithi064.stdout:332 trunc from 0xea8005a to 0x6b3b43a 2024-04-24T00:34:01.604 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.604 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.618 INFO:teuthology.orchestra.run.smithi064.stdout:335 write 0x7fa1259 thru 0x7fa8f28 (0x7cd0 bytes) 2024-04-24T00:34:01.647 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:01.647 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:01.647 INFO:teuthology.orchestra.run.smithi064.stdout:338 writesame 0xee261fe thru 0xee360d4 data_size 0x1(0xfed7 bytes) 2024-04-24T00:34:01.653 INFO:teuthology.orchestra.run.smithi064.stdout:339 trunc from 0xee360d5 to 0xae5636a 2024-04-24T00:34:01.667 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.667 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:01.680 INFO:teuthology.orchestra.run.smithi064.stdout:341 trunc from 0xae5636a to 0x2f2cd90 2024-04-24T00:34:01.680 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:01.676+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:01.699 INFO:teuthology.orchestra.run.smithi064.stdout:343 trunc from 0x2f2cd90 to 0x14da0ed 2024-04-24T00:34:01.714 INFO:teuthology.orchestra.run.smithi064.stdout:347 read 0x1173ced thru 0x117d742 (0x9a56 bytes) 2024-04-24T00:34:01.739 INFO:teuthology.orchestra.run.smithi064.stdout:349 write 0xc1b22ac thru 0xc1bb3ca (0x911f bytes) 2024-04-24T00:34:01.746 INFO:teuthology.orchestra.run.smithi064.stdout:350 read 0x9f12e65 thru 0x9f205d4 (0xd770 bytes) 2024-04-24T00:34:01.746 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.746 INFO:teuthology.orchestra.run.smithi064.stdout:353 read 0xa2fbafb thru 0xa3083b9 (0xc8bf bytes) 2024-04-24T00:34:01.746 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.746 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.746 INFO:teuthology.orchestra.run.smithi064.stdout:356 write 0x5b6b174 thru 0x5b7a1fb (0xf088 bytes) 2024-04-24T00:34:01.755 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.755 INFO:teuthology.orchestra.run.smithi064.stdout:360 writesame 0xbd25786 thru 0xbd26516 data_size 0x1(0xd91 bytes) 2024-04-24T00:34:01.761 INFO:teuthology.orchestra.run.smithi064.stdout:361 writesame 0x8250c4c thru 0x825326f data_size 0x2(0x2624 bytes) 2024-04-24T00:34:01.778 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:01.772+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:01.791 INFO:teuthology.orchestra.run.smithi064.stdout:362 trunc from 0xc1bb3cb to 0x34fddb8 2024-04-24T00:34:01.801 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:01.801 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:01.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:01.823 INFO:teuthology.orchestra.run.smithi064.stdout:364 read 0x131e1d thru 0x13d5c5 (0xb7a9 bytes) 2024-04-24T00:34:01.827 INFO:teuthology.orchestra.run.smithi064.stdout:366 trunc from 0x34fddb8 to 0x5491f3e 2024-04-24T00:34:01.837 INFO:teuthology.orchestra.run.smithi064.stdout:368 read 0x4f4bbd0 thru 0x4f5a3fa (0xe82b bytes) 2024-04-24T00:34:01.838 INFO:teuthology.orchestra.run.smithi064.stdout:370 read 0x50a0f2d thru 0x50aeaea (0xdbbe bytes) 2024-04-24T00:34:01.864 INFO:teuthology.orchestra.run.smithi064.stdout:371 writesame 0xdfdede4 thru 0xdfeccd5 data_size 0x1(0xdef2 bytes) 2024-04-24T00:34:01.864 INFO:teuthology.orchestra.run.smithi064.stdout:372 punch from 0xbb3464c to 0xbb3dafb, (0x94af bytes) 2024-04-24T00:34:01.864 INFO:teuthology.orchestra.run.smithi064.stdout:375 write 0x96c3c29 thru 0x96d35f1 (0xf9c9 bytes) 2024-04-24T00:34:01.872 INFO:teuthology.orchestra.run.smithi064.stdout:376 read 0x6f661e8 thru 0x6f6f46d (0x9286 bytes) 2024-04-24T00:34:01.873 INFO:teuthology.orchestra.run.smithi064.stdout:377 punch from 0x8c7ce8f to 0x8c83a1b, (0x6b8c bytes) 2024-04-24T00:34:01.873 INFO:teuthology.orchestra.run.smithi064.stdout:379 write 0xb45a8b2 thru 0xb45d92e (0x307d bytes) 2024-04-24T00:34:01.879 INFO:teuthology.orchestra.run.smithi064.stdout:380 write 0x28b28d2 thru 0x28bf795 (0xcec4 bytes) 2024-04-24T00:34:01.879 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:01.876+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:01.887 INFO:teuthology.orchestra.run.smithi064.stdout:381 read 0x5463be7 thru 0x546ca45 (0x8e5f bytes) 2024-04-24T00:34:01.887 INFO:teuthology.orchestra.run.smithi064.stdout:383 write 0x5c63ecd thru 0x5c6c68c (0x87c0 bytes) 2024-04-24T00:34:01.894 INFO:teuthology.orchestra.run.smithi064.stdout:384 write 0x90d3884 thru 0x90d57c4 (0x1f41 bytes) 2024-04-24T00:34:01.899 INFO:teuthology.orchestra.run.smithi064.stdout:385 read 0x9bb8278 thru 0x9bc2005 (0x9d8e bytes) 2024-04-24T00:34:01.899 INFO:teuthology.orchestra.run.smithi064.stdout:386 punch from 0x772b10b to 0x77319f7, (0x68ec bytes) 2024-04-24T00:34:01.905 INFO:teuthology.orchestra.run.smithi064.stdout:389 trunc from 0xdfeccd6 to 0xb8f02d4 2024-04-24T00:34:01.920 INFO:teuthology.orchestra.run.smithi064.stdout:390 write 0xaa7b1e5 thru 0xaa7f6a0 (0x44bc bytes) 2024-04-24T00:34:01.927 INFO:teuthology.orchestra.run.smithi064.stdout:392 writesame 0x95f94cc thru 0x96007a3 data_size 0x4b(0x72d8 bytes) 2024-04-24T00:34:01.931 INFO:teuthology.orchestra.run.smithi064.stdout:394 punch from 0x9dcde92 to 0x9dd6bcd, (0x8d3b bytes) 2024-04-24T00:34:01.931 INFO:teuthology.orchestra.run.smithi064.stdout:395 read 0xa5817d4 thru 0xa588bb9 (0x73e6 bytes) 2024-04-24T00:34:01.932 INFO:teuthology.orchestra.run.smithi064.stdout:396 writesame 0x363ddb9 thru 0x363ebb3 data_size 0x1(0xdfb bytes) 2024-04-24T00:34:01.937 INFO:teuthology.orchestra.run.smithi064.stdout:397 read 0x785b879 thru 0x7862f1c (0x76a4 bytes) 2024-04-24T00:34:01.937 INFO:teuthology.orchestra.run.smithi064.stdout:400 read 0x973eac1 thru 0x974a6d1 (0xbc11 bytes) 2024-04-24T00:34:01.938 INFO:teuthology.orchestra.run.smithi064.stdout:401 write 0x9dda713 thru 0x9de4fe1 (0xa8cf bytes) 2024-04-24T00:34:01.945 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:01.946 INFO:teuthology.orchestra.run.smithi064.stdout:403 read 0x957892d thru 0x9579e83 (0x1557 bytes) 2024-04-24T00:34:01.946 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:01.946 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:01.947 INFO:teuthology.orchestra.run.smithi064.stdout:404 writesame 0x1d06ac5 thru 0x1d10de3 data_size 0x1(0xa31f bytes) 2024-04-24T00:34:01.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:02.003 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:02.000+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:02.082 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:02.076+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:02.093 INFO:teuthology.orchestra.run.smithi064.stdout:405 trunc from 0xb8f02d4 to 0x61ae8dd 2024-04-24T00:34:02.123 INFO:teuthology.orchestra.run.smithi064.stdout:406 read 0x38d54c4 thru 0x38db256 (0x5d93 bytes) 2024-04-24T00:34:02.124 INFO:teuthology.orchestra.run.smithi064.stdout:407 punch from 0x4d7a331 to 0x4d7e541, (0x4210 bytes) 2024-04-24T00:34:02.130 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:02.130 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:02.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:02.158 INFO:teuthology.orchestra.run.smithi064.stdout:409 write 0xe3e9cce thru 0xe3f12de (0x7611 bytes) 2024-04-24T00:34:02.164 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.165 INFO:teuthology.orchestra.run.smithi064.stdout:411 punch from 0xd88d3eb to 0xd89b480, (0xe095 bytes) 2024-04-24T00:34:02.165 INFO:teuthology.orchestra.run.smithi064.stdout:413 punch from 0xda3a364 to 0xda45fe2, (0xbc7e bytes) 2024-04-24T00:34:02.165 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.170 INFO:teuthology.orchestra.run.smithi064.stdout:416 trunc from 0xe3f12df to 0xbe73650 2024-04-24T00:34:02.182 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:02.176+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:02.192 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:420 read 0xc634c1 thru 0xc6b00d (0x7b4d bytes) 2024-04-24T00:34:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:421 writesame 0x619bfa7 thru 0x61a8a86 data_size 0x30(0xcae0 bytes) 2024-04-24T00:34:02.198 INFO:teuthology.orchestra.run.smithi064.stdout:422 write 0x88bc891 thru 0x88c5cd0 (0x9440 bytes) 2024-04-24T00:34:02.205 INFO:teuthology.orchestra.run.smithi064.stdout:423 read 0x24dd72b thru 0x24e28ff (0x51d5 bytes) 2024-04-24T00:34:02.218 INFO:teuthology.orchestra.run.smithi064.stdout:424 write 0xc230cc1 thru 0xc23b7cb (0xab0b bytes) 2024-04-24T00:34:02.225 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0xe0ffb48 2024-04-24T00:34:02.231 INFO:teuthology.orchestra.run.smithi064.stdout:425 trunc from 0xc23b7cc to 0xe0ffb48 2024-04-24T00:34:02.243 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.243 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.243 INFO:teuthology.orchestra.run.smithi064.stdout:428 read 0xc8ff81d thru 0xc908343 (0x8b27 bytes) 2024-04-24T00:34:02.244 INFO:teuthology.orchestra.run.smithi064.stdout:430 write 0xc3d87c6 thru 0xc3e5ab9 (0xd2f4 bytes) 2024-04-24T00:34:02.257 INFO:teuthology.orchestra.run.smithi064.stdout:431 writesame 0xc769824 thru 0xc76e8d2 data_size 0x55(0x50af bytes) 2024-04-24T00:34:02.283 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:02.276+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:02.287 INFO:teuthology.orchestra.run.smithi064.stdout:432 trunc from 0xe0ffb48 to 0x3cc658b 2024-04-24T00:34:02.310 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:02.310 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:02.323 INFO:teuthology.orchestra.run.smithi064.stdout:435 trunc from 0x3cc658b to 0x5a295b1 2024-04-24T00:34:02.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:02.337 INFO:teuthology.orchestra.run.smithi064.stdout:437 trunc from 0x5a295b1 to 0x51ae5eb 2024-04-24T00:34:02.365 INFO:teuthology.orchestra.run.smithi064.stdout:438 write 0x5a68752 thru 0x5a6e07d (0x592c bytes) 2024-04-24T00:34:02.369 INFO:teuthology.orchestra.run.smithi064.stdout:439 writesame 0x1c86df6 thru 0x1c88773 data_size 0xd(0x197e bytes) 2024-04-24T00:34:02.385 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:02.380+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:02.406 INFO:teuthology.orchestra.run.smithi064.stdout:440 write 0xdb14c34 thru 0xdb1ee45 (0xa212 bytes) 2024-04-24T00:34:02.414 INFO:teuthology.orchestra.run.smithi064.stdout:441 read 0x95ec90b thru 0x95f667d (0x9d73 bytes) 2024-04-24T00:34:02.415 INFO:teuthology.orchestra.run.smithi064.stdout:446 write 0x6be1499 thru 0x6bef4ac (0xe014 bytes) 2024-04-24T00:34:02.425 INFO:teuthology.orchestra.run.smithi064.stdout:447 write 0x65bba5f thru 0x65c5053 (0x95f5 bytes) 2024-04-24T00:34:02.434 INFO:teuthology.orchestra.run.smithi064.stdout:449 writesame 0x90cf4e7 thru 0x90db254 data_size 0x1(0xbd6e bytes) 2024-04-24T00:34:02.460 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:02.460 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:02.479 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-04-24T00:34:02.486 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:02.480+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:02.489 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:02.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-24T00:34:02.563 INFO:teuthology.orchestra.run.smithi064.stdout:450 punch from 0xaef902b to 0xaefac37, (0x1c0c bytes) 2024-04-24T00:34:02.563 INFO:teuthology.orchestra.run.smithi064.stdout:451 read 0x3d2f1e6 thru 0x3d33c18 (0x4a33 bytes) 2024-04-24T00:34:02.563 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.563 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.563 INFO:teuthology.orchestra.run.smithi064.stdout:456 read 0x10a4ffd thru 0x10b0d00 (0xbd04 bytes) 2024-04-24T00:34:02.564 INFO:teuthology.orchestra.run.smithi064.stdout:457 write 0x493aad3 thru 0x4944ff1 (0xa51f bytes) 2024-04-24T00:34:02.568 INFO:teuthology.orchestra.run.smithi064.stdout:458 punch from 0x5adc94d to 0x5ae3830, (0x6ee3 bytes) 2024-04-24T00:34:02.587 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:02.580+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:02.589 INFO:teuthology.orchestra.run.smithi064.stdout:459 trunc from 0xdb1ee46 to 0x3edafe5 2024-04-24T00:34:02.610 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:02.610 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:02.625 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:02.626 INFO:teuthology.orchestra.run.smithi064.stdout:460 writesame 0x5a7dde9 thru 0x5a80788 data_size 0x30(0x29a0 bytes) 2024-04-24T00:34:02.630 INFO:teuthology.orchestra.run.smithi064.stdout:461 read 0x485cfea thru 0x4861e8d (0x4ea4 bytes) 2024-04-24T00:34:02.631 INFO:teuthology.orchestra.run.smithi064.stdout:462 read 0x24f0110 thru 0x24fef0c (0xedfd bytes) 2024-04-24T00:34:02.648 INFO:teuthology.orchestra.run.smithi064.stdout:464 write 0x8f156d9 thru 0x8f1820a (0x2b32 bytes) 2024-04-24T00:34:02.653 INFO:teuthology.orchestra.run.smithi064.stdout:465 write 0x6abd344 thru 0x6abe17c (0xe39 bytes) 2024-04-24T00:34:02.656 INFO:teuthology.orchestra.run.smithi064.stdout:466 punch from 0x7dbd65c to 0x7dc99ec, (0xc390 bytes) 2024-04-24T00:34:02.656 INFO:teuthology.orchestra.run.smithi064.stdout:467 write 0x69cc9b7 thru 0x69da3df (0xda29 bytes) 2024-04-24T00:34:02.674 INFO:teuthology.orchestra.run.smithi064.stdout:468 write 0x9118a83 thru 0x9120b87 (0x8105 bytes) 2024-04-24T00:34:02.680 INFO:teuthology.orchestra.run.smithi064.stdout:472 write 0x53711ae thru 0x53780bd (0x6f10 bytes) 2024-04-24T00:34:02.685 INFO:teuthology.orchestra.run.smithi064.stdout:473 punch from 0x3c5663f to 0x3c5abb1, (0x4572 bytes) 2024-04-24T00:34:02.685 INFO:teuthology.orchestra.run.smithi064.stdout:474 read 0x49262de thru 0x4933e60 (0xdb83 bytes) 2024-04-24T00:34:02.685 INFO:teuthology.orchestra.run.smithi064.stdout:475 write 0x89653e7 thru 0x896cf16 (0x7b30 bytes) 2024-04-24T00:34:02.688 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:02.684+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:02.692 INFO:teuthology.orchestra.run.smithi064.stdout:476 write 0x8b89d06 thru 0x8b96a82 (0xcd7d bytes) 2024-04-24T00:34:02.698 INFO:teuthology.orchestra.run.smithi064.stdout:477 read 0x570f425 thru 0x5714b32 (0x570e bytes) 2024-04-24T00:34:02.701 INFO:teuthology.orchestra.run.smithi064.stdout:478 trunc from 0x9120b88 to 0x7466ea0 2024-04-24T00:34:02.720 INFO:teuthology.orchestra.run.smithi064.stdout:479 read 0x41cef4a thru 0x41d6311 (0x73c8 bytes) 2024-04-24T00:34:02.729 INFO:teuthology.orchestra.run.smithi064.stdout:480 write 0x8edc811 thru 0x8edf8c8 (0x30b8 bytes) 2024-04-24T00:34:02.733 INFO:teuthology.orchestra.run.smithi064.stdout:481 write 0x6363506 thru 0x636c028 (0x8b23 bytes) 2024-04-24T00:34:02.736 INFO:teuthology.orchestra.run.smithi064.stdout:482 read 0x53571be thru 0x5359ce2 (0x2b25 bytes) 2024-04-24T00:34:02.736 INFO:teuthology.orchestra.run.smithi064.stdout:483 read 0x6ba78ce thru 0x6ba885d (0xf90 bytes) 2024-04-24T00:34:02.737 INFO:teuthology.orchestra.run.smithi064.stdout:484 read 0x28b8a0c thru 0x28c2526 (0x9b1b bytes) 2024-04-24T00:34:02.743 INFO:teuthology.orchestra.run.smithi064.stdout:485 trunc from 0x8edf8c9 to 0xceb1553 2024-04-24T00:34:02.751 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.753 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:02.753 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:02.763 INFO:teuthology.orchestra.run.smithi064.stdout:487 trunc from 0xceb1553 to 0x6c9d23e 2024-04-24T00:34:02.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:02.784 INFO:teuthology.orchestra.run.smithi064.stdout:488 read 0x48c230d thru 0x48d0fbc (0xecb0 bytes) 2024-04-24T00:34:02.785 INFO:teuthology.orchestra.run.smithi064.stdout:489 writesame 0x3b6be88 thru 0x3b760b4 data_size 0x15(0xa22d bytes) 2024-04-24T00:34:02.807 INFO:teuthology.orchestra.run.smithi064.stdout:490 write 0xa178a59 thru 0xa186cb1 (0xe259 bytes) 2024-04-24T00:34:02.811 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:02.808+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:02.823 INFO:teuthology.orchestra.run.smithi064.stdout:492 write 0xa94754a thru 0xa94b29d (0x3d54 bytes) 2024-04-24T00:34:02.830 INFO:teuthology.orchestra.run.smithi064.stdout:493 read 0x397b55b thru 0x398ab8b (0xf631 bytes) 2024-04-24T00:34:02.831 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.831 INFO:teuthology.orchestra.run.smithi064.stdout:496 writesame 0x7a655aa thru 0x7a69448 data_size 0x17(0x3e9f bytes) 2024-04-24T00:34:02.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:02.836 INFO:teuthology.orchestra.run.smithi064.stdout:497 read 0x3ef23a3 thru 0x3ef761b (0x5279 bytes) 2024-04-24T00:34:02.837 INFO:teuthology.orchestra.run.smithi064.stdout:498 write 0x72e1b1e thru 0x72eeb67 (0xd04a bytes) 2024-04-24T00:34:02.845 INFO:teuthology.orchestra.run.smithi064.stdout:499 read 0x1c856fc thru 0x1c91fd4 (0xc8d9 bytes) 2024-04-24T00:34:02.861 INFO:teuthology.orchestra.run.smithi064.stdout:500 writesame 0xe8708eb thru 0xe87d809 data_size 0x1(0xcf1f bytes) 2024-04-24T00:34:02.861 INFO:teuthology.orchestra.run.smithi064.stdout:501 write 0x37b11ca thru 0x37bb39a (0xa1d1 bytes) 2024-04-24T00:34:02.864 INFO:teuthology.orchestra.run.smithi064.stdout:502 write 0x5852855 thru 0x585f3f8 (0xcba4 bytes) 2024-04-24T00:34:02.873 INFO:teuthology.orchestra.run.smithi064.stdout:503 read 0x9cf4fea thru 0x9cf80b5 (0x30cc bytes) 2024-04-24T00:34:02.874 INFO:teuthology.orchestra.run.smithi064.stdout:504 punch from 0xd41462b to 0xd41e775, (0xa14a bytes) 2024-04-24T00:34:02.874 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.874 INFO:teuthology.orchestra.run.smithi064.stdout:508 write 0x8976cbe thru 0x8977f85 (0x12c8 bytes) 2024-04-24T00:34:02.878 INFO:teuthology.orchestra.run.smithi064.stdout:509 writesame 0x9301030 thru 0x930180e data_size 0xd(0x7df bytes) 2024-04-24T00:34:02.881 INFO:teuthology.orchestra.run.smithi064.stdout:510 write 0xaae467b thru 0xaaed995 (0x931b bytes) 2024-04-24T00:34:02.884 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.884 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:02.890 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:02.884+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:02.895 INFO:teuthology.orchestra.run.smithi064.stdout:514 writesame 0xec0af5b thru 0xec0dc1e data_size 0x14(0x2cc4 bytes) 2024-04-24T00:34:02.895 INFO:teuthology.orchestra.run.smithi064.stdout:515 read 0x528e800 thru 0x5290613 (0x1e14 bytes) 2024-04-24T00:34:02.896 INFO:teuthology.orchestra.run.smithi064.stdout:518 punch from 0xbb572b8 to 0xbb61326, (0xa06e bytes) 2024-04-24T00:34:02.896 INFO:teuthology.orchestra.run.smithi064.stdout:519 write 0x34c2b37 thru 0x34cd110 (0xa5da bytes) 2024-04-24T00:34:02.901 INFO:teuthology.orchestra.run.smithi064.stdout:521 clone 2 order 21 su 65536 sc 2 2024-04-24T00:34:02.937 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:02.937 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:02.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:02.984 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:02.984 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:02.991 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:02.984+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:03.001 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:03.001 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:03.017 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:03.018 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:03.034 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:03.034 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:03.052 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:03.072 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:03.072 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:03.092 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:03.088+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:03.093 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:03.093 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:03.112 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:03.123 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:557 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:58.502655+0000", 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.614904449, 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024169009999999999, 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 557 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502655+0000", 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502655+0000", 2024-04-24T00:34:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502656+0000", 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.313e-06 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502662+0000", 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5300000000000004e-06 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502673+0000", 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.169e-05 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502679+0000", 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7880000000000003e-06 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.502782+0000", 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102835 2024-04-24T00:34:03.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.503009+0000", 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022671799999999999 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.503357+0000", 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000348334 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.504504+0000", 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011466409999999999 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.504870+0000", 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036630099999999997 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.504957+0000", 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7496000000000004e-05 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:58.505072+0000", 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114255 2024-04-24T00:34:03.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:580 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.449954+0000", 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6676051799999998, 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057157029999999999, 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.130 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 580 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.449954+0000", 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.449954+0000", 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.449957+0000", 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.044e-06 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.449971+0000", 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.450001+0000", 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9624e-05 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.450021+0000", 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.022e-05 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.450151+0000", 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013028900000000001 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:03.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.450163+0000", 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.183e-05 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.452964+0000", 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028005539999999998 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.453114+0000", 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015016300000000001 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.453867+0000", 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00075320400000000005 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.455260+0000", 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001393285 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.455508+0000", 2024-04-24T00:34:03.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024713799999999999 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.455592+0000", 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4062999999999998e-05 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.455670+0000", 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8022999999999995e-05 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.136 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:584 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.set_protection_status in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.461569+0000", 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6559902520000001, 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002681606, 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 584 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.461569+0000", 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.461569+0000", 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.461571+0000", 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3300000000000001e-06 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.461583+0000", 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1783000000000001e-05 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.461612+0000", 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9183000000000002e-05 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.461627+0000", 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5153e-05 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.461780+0000", 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015312600000000001 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.462039+0000", 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025844399999999998 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.462705+0000", 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000666459 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.463996+0000", 2024-04-24T00:34:03.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012905200000000001 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.464083+0000", 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7422000000000004e-05 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.464169+0000", 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5890000000000006e-05 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.464251+0000", 2024-04-24T00:34:03.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1296000000000002e-05 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:681 3.10 3:0910cee1:::rbd_data.115e92aeb10c.0000000000000033:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.473210+0000", 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6443490450000002, 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003198448, 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.142 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 681 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473210+0000", 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473210+0000", 2024-04-24T00:34:03.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473318+0000", 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108095 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473333+0000", 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4414e-05 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473370+0000", 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7299999999999999e-05 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473392+0000", 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2342999999999999e-05 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.476242+0000", 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028498410000000001 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.476409+0000", 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000166455 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:697 3.10 3:0910cee1:::rbd_data.115e92aeb10c.0000000000000033:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.473449+0000", 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6441106109999999, 2024-04-24T00:34:03.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031226700000000001, 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 697 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473449+0000", 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473449+0000", 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473450+0000", 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6500000000000001e-06 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473458+0000", 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8469999999999996e-06 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473479+0000", 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1423000000000001e-05 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473497+0000", 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.755e-05 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.476450+0000", 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029533269999999999 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.476571+0000", 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012087300000000001 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:713 3.10 3:0910cee1:::rbd_data.115e92aeb10c.0000000000000033:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.150 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.473524+0000", 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.644035454, 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003197245, 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 713 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473524+0000", 2024-04-24T00:34:03.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473524+0000", 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473525+0000", 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473534+0000", 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6230000000000003e-06 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473563+0000", 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8935999999999999e-05 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473583+0000", 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.056e-05 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.476607+0000", 2024-04-24T00:34:03.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003023764 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.476721+0000", 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011380200000000001 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:729 3.10 3:0910cee1:::rbd_data.115e92aeb10c.0000000000000033:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.473609+0000", 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6439498219999997, 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003256985, 2024-04-24T00:34:03.154 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 729 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473609+0000", 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473609+0000", 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473611+0000", 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4699999999999999e-06 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473617+0000", 2024-04-24T00:34:03.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6599999999999998e-06 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473649+0000", 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1504000000000002e-05 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.473664+0000", 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5319e-05 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.476757+0000", 2024-04-24T00:34:03.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030924400000000001 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.476866+0000", 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109592 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:816 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.538350+0000", 2024-04-24T00:34:03.158 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5792091080000001, 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030985489999999999, 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 816 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538350+0000", 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538350+0000", 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538352+0000", 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538360+0000", 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9130000000000003e-06 2024-04-24T00:34:03.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538381+0000", 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1268999999999999e-05 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538392+0000", 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1257000000000001e-05 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538549+0000", 2024-04-24T00:34:03.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015655899999999999 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538696+0000", 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014718199999999999 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.538988+0000", 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029188100000000002 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.540171+0000", 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011823980000000001 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.541293+0000", 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011225740000000001 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.541381+0000", 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7448999999999996e-05 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.541449+0000", 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8087e-05 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:820 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.snapshot_trash_add in=21b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.544177+0000", 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5733825870000002, 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026263139999999998, 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 820 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544177+0000", 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544177+0000", 2024-04-24T00:34:03.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544178+0000", 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544183+0000", 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3329999999999996e-06 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544216+0000", 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2756999999999998e-05 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544226+0000", 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8300000000000008e-06 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544689+0000", 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00046307199999999997 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544823+0000", 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013422600000000001 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.545329+0000", 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00050642299999999999 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.546408+0000", 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010782019999999999 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.546611+0000", 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020362099999999999 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.546702+0000", 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0370000000000004e-05 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.546803+0000", 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101396 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:821 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:59.544689+0000", 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5728704570000001, 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024724040000000001, 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 821 2024-04-24T00:34:03.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544689+0000", 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544689+0000", 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544691+0000", 2024-04-24T00:34:03.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.866e-06 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544698+0000", 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5399999999999998e-06 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544719+0000", 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.124e-05 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.544730+0000", 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1046999999999999e-05 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.545928+0000", 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011971810000000001 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.546145+0000", 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021742500000000001 2024-04-24T00:34:03.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.546869+0000", 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072382399999999997 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.546985+0000", 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116029 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:59.547161+0000", 2024-04-24T00:34:03.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017625200000000001 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:827 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.461004+0000", 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.6565548469999998, 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069575740000000002, 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.175 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 827 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.461004+0000", 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.461004+0000", 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.461007+0000", 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8899999999999999e-06 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.461020+0000", 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2573e-05 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.461049+0000", 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8974000000000001e-05 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.461073+0000", 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4409999999999998e-05 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.461216+0000", 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000142705 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.461254+0000", 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8269999999999998e-05 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.465217+0000", 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039632590000000002 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.465396+0000", 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178622 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.466022+0000", 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00062620199999999997 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.467658+0000", 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001635364 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.467798+0000", 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014020600000000001 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.467890+0000", 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1796e-05 2024-04-24T00:34:03.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.467962+0000", 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2303000000000002e-05 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:831 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.476376+0000", 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.6411831819999998, 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002950051, 2024-04-24T00:34:03.181 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 831 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.476376+0000", 2024-04-24T00:34:03.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.476376+0000", 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.476380+0000", 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9129999999999996e-06 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.476394+0000", 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3740000000000001e-05 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.476433+0000", 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9579999999999997e-05 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.476457+0000", 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3300000000000001e-05 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.476622+0000", 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016510900000000001 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.477048+0000", 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042647600000000001 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.477558+0000", 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051035999999999996 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.478794+0000", 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012357469999999999 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479168+0000", 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037343000000000001 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479256+0000", 2024-04-24T00:34:03.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8417000000000004e-05 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479326+0000", 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9979000000000002e-05 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:833 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [watch unwatch cookie 140203905902416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.479641+0000", 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.637918167, 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025041009999999999, 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.187 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 833 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479641+0000", 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479641+0000", 2024-04-24T00:34:03.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479642+0000", 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1340000000000001e-06 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479655+0000", 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2553e-05 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479668+0000", 2024-04-24T00:34:03.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3315999999999999e-05 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.479680+0000", 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2357e-05 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480085+0000", 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040469699999999998 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480222+0000", 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013653499999999999 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480445+0000", 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022389900000000001 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.481453+0000", 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010078789999999999 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.481905+0000", 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045138300000000003 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.481989+0000", 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4169e-05 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.482145+0000", 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156179 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.192 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:834 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.480187+0000", 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.6373722310000001, 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025053440000000001, 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 834 2024-04-24T00:34:03.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480187+0000", 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480187+0000", 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480189+0000", 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480198+0000", 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0499999999999997e-06 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480220+0000", 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1843000000000001e-05 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480231+0000", 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1229999999999999e-05 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.480899+0000", 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066781499999999999 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.481010+0000", 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111049 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.482254+0000", 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012445710000000001 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.482462+0000", 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020787799999999999 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.482692+0000", 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022992800000000001 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:848 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [watch watch cookie 140203905876320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.632203+0000", 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.485355905, 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025054579999999999, 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 848 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.632203+0000", 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.632203+0000", 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.632205+0000", 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.097e-06 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.632217+0000", 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1396999999999999e-05 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.632251+0000", 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3745999999999998e-05 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.632270+0000", 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9077e-05 2024-04-24T00:34:03.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.632390+0000", 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120839 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.632619+0000", 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022903800000000001 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.633053+0000", 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043368999999999998 2024-04-24T00:34:03.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.634362+0000", 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001309197 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.634507+0000", 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144158 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.634599+0000", 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2310000000000002e-05 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.634709+0000", 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010990899999999999 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:856 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.651422+0000", 2024-04-24T00:34:03.203 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.4661368590000001, 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027312579999999999, 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 856 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.651422+0000", 2024-04-24T00:34:03.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.651422+0000", 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.651425+0000", 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2929999999999999e-06 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.651436+0000", 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.131e-05 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.651467+0000", 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1205999999999999e-05 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.651482+0000", 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4783000000000001e-05 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.651633+0000", 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000151249 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.651891+0000", 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025798800000000001 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.652449+0000", 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055808300000000002 2024-04-24T00:34:03.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.653873+0000", 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001423945 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.653915+0000", 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2012999999999999e-05 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.654040+0000", 2024-04-24T00:34:03.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012525899999999999 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.654154+0000", 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011312899999999999 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:866 3.c e28/21 3:36b546fb:::rbd_data.115eb9c76af6.0000000000000004:head v 28'109, mlcod=26'107)", 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.669726+0000", 2024-04-24T00:34:03.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.4478336409999999, 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042492739999999999, 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.669726+0000", 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.669726+0000", 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.669726+0000", 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8599999999999998e-07 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.669879+0000", 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152743 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.669902+0000", 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.283e-05 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.669919+0000", 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7436e-05 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.670025+0000", 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105889 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.670065+0000", 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0157000000000002e-05 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.673897+0000", 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038322640000000002 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.673975+0000", 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7368999999999997e-05 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:875 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.700323+0000", 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.417235824, 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026019540000000001, 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 875 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.214 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.700323+0000", 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.700323+0000", 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.700326+0000", 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6740000000000001e-06 2024-04-24T00:34:03.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.700338+0000", 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2293000000000001e-05 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.700377+0000", 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8835999999999999e-05 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.700392+0000", 2024-04-24T00:34:03.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4766999999999999e-05 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.700605+0000", 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021293199999999999 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.700795+0000", 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190352 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.701371+0000", 2024-04-24T00:34:03.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00057610199999999999 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.702577+0000", 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00120612 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.702753+0000", 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175079 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.702841+0000", 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8923000000000002e-05 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.702925+0000", 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3875999999999995e-05 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:885 3.10 3:092b2319:::rbd_header.115eb9c76af6:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.720965+0000", 2024-04-24T00:34:03.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.3965938759999998, 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024362110000000002, 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 885 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.720965+0000", 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.720965+0000", 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.720968+0000", 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3630000000000001e-06 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.720980+0000", 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1917e-05 2024-04-24T00:34:03.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.721010+0000", 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0539e-05 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.721025+0000", 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5282999999999999e-05 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.721158+0000", 2024-04-24T00:34:03.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013239600000000001 2024-04-24T00:34:03.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.721333+0000", 2024-04-24T00:34:03.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175592 2024-04-24T00:34:03.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.721643+0000", 2024-04-24T00:34:03.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030929800000000001 2024-04-24T00:34:03.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.723049+0000", 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014066059999999999 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.723100+0000", 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0519000000000002e-05 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.723259+0000", 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015944200000000001 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.723402+0000", 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000142256 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:908 3.c e28/21 3:36b546fb:::rbd_data.115eb9c76af6.0000000000000004:head v 28'110, mlcod=27'108)", 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:00.760423+0000", 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.3571366409999999, 2024-04-24T00:34:03.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025791099999999999, 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.760423+0000", 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.760423+0000", 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.760424+0000", 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7740000000000001e-06 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.760475+0000", 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1062e-05 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.760504+0000", 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8574000000000001e-05 2024-04-24T00:34:03.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.760517+0000", 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.344e-05 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.760643+0000", 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012577899999999999 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.760681+0000", 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7855999999999998e-05 2024-04-24T00:34:03.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.762913+0000", 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022316060000000001 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:00.763002+0000", 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9018999999999996e-05 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.231 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:03.233 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:03.228+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:03.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:03.334 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:03.328+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:03.402 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:03.402 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:03.402 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:03.402 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:03.416 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:03.435 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:03.428+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:03.476 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.7 to 1 (10000) 2024-04-24T00:34:03.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:34:03.505 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T00:34:03.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:34:03.536 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:03.532+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:03.543 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:03.543 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:03.543 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:03.543 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:03.543 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:03.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:03.643 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:03.640+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.794232247, 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.635694761, 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011455702, 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 37 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:34:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084943+0000", 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084944+0000", 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1999999999999999e-07 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084950+0000", 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6200000000000001e-06 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084964+0000", 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3186999999999999e-05 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.084971+0000", 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3289999999999996e-06 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094143+0000", 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091718640000000001 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094289+0000", 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014654499999999999 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.094480+0000", 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019091799999999999 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.095845+0000", 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00136509 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096272+0000", 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042710300000000002 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096345+0000", 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2243000000000006e-05 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.096399+0000", 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4082999999999997e-05 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.340913061999998, 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013518105000000001, 2024-04-24T00:34:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 110 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379725+0000", 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379726+0000", 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5770000000000001e-06 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379738+0000", 2024-04-24T00:34:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1552999999999999e-05 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379758+0000", 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9636e-05 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379773+0000", 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4970000000000001e-05 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388655+0000", 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088828440000000009 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388781+0000", 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012547200000000001 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.388993+0000", 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021258200000000001 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.392907+0000", 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039138669999999997 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393039+0000", 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013181900000000001 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393112+0000", 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2853000000000002e-05 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393243+0000", 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130932 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.340842063, 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013665012000000001, 2024-04-24T00:34:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 111 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379796+0000", 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379797+0000", 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.173e-06 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379806+0000", 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3370000000000006e-06 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379825+0000", 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8519999999999999e-05 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379839+0000", 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.401e-05 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390285+0000", 2024-04-24T00:34:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010446162 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390412+0000", 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012749899999999999 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.390614+0000", 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020195799999999999 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393015+0000", 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024003789999999998 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393268+0000", 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025304800000000002 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393356+0000", 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8613e-05 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393461+0000", 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104313 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.340771557, 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019315533999999999, 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.749 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 112 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379868+0000", 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.274e-06 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380096+0000", 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022802799999999999 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380146+0000", 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0358999999999997e-05 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380164+0000", 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7713000000000001e-05 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391250+0000", 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011085928 2024-04-24T00:34:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391383+0000", 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013310899999999999 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391576+0000", 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019287899999999999 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395703+0000", 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041278740000000001 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398947+0000", 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032432149999999998 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399035+0000", 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8635999999999996e-05 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399182+0000", 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014651900000000001 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.340426105999999, 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019210844000000001, 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 113 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380212+0000", 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380213+0000", 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5829999999999999e-06 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380225+0000", 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1603999999999999e-05 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380244+0000", 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9170000000000001e-05 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380258+0000", 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4382999999999999e-05 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393574+0000", 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013315726999999999 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393703+0000", 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012839200000000001 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.393904+0000", 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000201805 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398808+0000", 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049032820000000001 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399200+0000", 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039261 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399310+0000", 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399423+0000", 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011273900000000001 2024-04-24T00:34:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.340349669999998, 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019317091000000002, 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 114 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380289+0000", 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.173e-06 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380300+0000", 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-05 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380324+0000", 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3047000000000001e-05 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380338+0000", 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4545999999999999e-05 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394634+0000", 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014296233 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394760+0000", 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012605900000000001 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394953+0000", 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019237900000000001 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398924+0000", 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039709589999999996 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399439+0000", 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051536599999999996 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399507+0000", 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8065999999999995e-05 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399605+0000", 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8133000000000004e-05 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.340236208, 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020360119999999999, 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 115 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380404+0000", 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:34:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380546+0000", 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014221200000000001 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380561+0000", 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5696e-05 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380572+0000", 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0353999999999999e-05 2024-04-24T00:34:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395831+0000", 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015259435 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395982+0000", 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150736 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.396234+0000", 2024-04-24T00:34:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000252334 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399712+0000", 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034776529999999998 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400554+0000", 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00084231699999999998 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400634+0000", 2024-04-24T00:34:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9270000000000005e-05 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400762+0000", 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012814900000000001 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.340046372, 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020559280999999999, 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 116 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:34:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380593+0000", 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380600+0000", 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0160000000000003e-06 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380614+0000", 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4317e-05 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380622+0000", 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.397613+0000", 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016991532 2024-04-24T00:34:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.397762+0000", 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149228 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398030+0000", 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026737499999999998 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400774+0000", 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027444959999999999 2024-04-24T00:34:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400968+0000", 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193358 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401045+0000", 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7440000000000004e-05 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401151+0000", 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105719 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.340003512999999, 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024520601, 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 117 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380635+0000", 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380642+0000", 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7700000000000004e-06 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380654+0000", 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1686e-05 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380666+0000", 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2150000000000001e-05 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399787+0000", 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019121461999999999 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399921+0000", 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134043 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400127+0000", 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020607500000000001 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401186+0000", 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001058738 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.404952+0000", 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037661779999999998 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405042+0000", 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9889999999999995e-05 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405155+0000", 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112812 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.83130907, 2024-04-24T00:34:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013799176999999999, 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 133 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889329+0000", 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889330+0000", 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.339e-06 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889338+0000", 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9610000000000005e-06 2024-04-24T00:34:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889352+0000", 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889361+0000", 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8100000000000004e-06 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891442+0000", 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020808860000000001 2024-04-24T00:34:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891501+0000", 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9314999999999997e-05 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891579+0000", 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7163000000000006e-05 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895799+0000", 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042201670000000004 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903071+0000", 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072725380000000003 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903101+0000", 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9828e-05 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903128+0000", 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6747e-05 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.831249894999999, 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014011816999999999, 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 134 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889388+0000", 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889389+0000", 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3990000000000001e-06 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889469+0000", 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9915999999999999e-05 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889482+0000", 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3215e-05 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889491+0000", 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8179999999999993e-06 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891840+0000", 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023490400000000002 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891898+0000", 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8031000000000001e-05 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.891977+0000", 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8512999999999999e-05 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895803+0000", 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00382593 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903233+0000", 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074298970000000004 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903329+0000", 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6718999999999993e-05 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903400+0000", 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0339000000000005e-05 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.831110781, 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013963768999999999, 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 135 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889527+0000", 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889528+0000", 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1100000000000004e-07 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889534+0000", 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3539999999999996e-06 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889545+0000", 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0684e-05 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889553+0000", 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2770000000000005e-06 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892602+0000", 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030488920000000001 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892651+0000", 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8905e-05 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.892729+0000", 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896463+0000", 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037338810000000001 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903439+0000", 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069760079999999997 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903468+0000", 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9349e-05 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903491+0000", 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2555999999999999e-05 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.831064959999999, 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01413908, 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 136 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889573+0000", 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889574+0000", 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1399999999999996e-07 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889579+0000", 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7810000000000002e-06 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889589+0000", 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3889999999999994e-06 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889597+0000", 2024-04-24T00:34:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7950000000000008e-06 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893012+0000", 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003415408 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893065+0000", 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2932000000000001e-05 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893157+0000", 2024-04-24T00:34:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2213999999999995e-05 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896469+0000", 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033114849999999999 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903563+0000", 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070944959999999996 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903649+0000", 2024-04-24T00:34:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5653000000000001e-05 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903712+0000", 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3113000000000004e-05 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.831006287000001, 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014166239000000001, 2024-04-24T00:34:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 137 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889632+0000", 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889713+0000", 2024-04-24T00:34:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0840999999999994e-05 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889722+0000", 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7329999999999992e-06 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889728+0000", 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8189999999999997e-06 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893442+0000", 2024-04-24T00:34:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037142540000000002 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893489+0000", 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7271000000000002e-05 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.893566+0000", 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6858000000000002e-05 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896481+0000", 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002914551 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903750+0000", 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072693860000000001 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903776+0000", 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6146999999999999e-05 2024-04-24T00:34:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.903798+0000", 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1478999999999999e-05 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.830896246999998, 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015393184000000001, 2024-04-24T00:34:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 138 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889742+0000", 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1200000000000003e-07 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889745+0000", 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3759999999999999e-06 2024-04-24T00:34:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889751+0000", 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2070000000000003e-06 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889755+0000", 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0230000000000003e-06 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894238+0000", 2024-04-24T00:34:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044834610000000002 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894288+0000", 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9434000000000001e-05 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894364+0000", 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5937000000000002e-05 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896492+0000", 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021286870000000002 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905091+0000", 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085987419999999995 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905116+0000", 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4752e-05 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905135+0000", 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9052999999999999e-05 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.830870595, 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015493453000000001, 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 139 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889767+0000", 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889768+0000", 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889774+0000", 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0050000000000001e-06 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889781+0000", 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7090000000000002e-06 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889785+0000", 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8489999999999999e-06 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894637+0000", 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048518320000000004 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894687+0000", 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0102000000000002e-05 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.894765+0000", 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7509000000000006e-05 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.896500+0000", 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017353259999999999 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905183+0000", 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086828540000000003 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905208+0000", 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5622999999999998e-05 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.905261+0000", 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2244999999999999e-05 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.830844456000001, 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018149166000000001, 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889793+0000", 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889794+0000", 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3200000000000001e-07 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889854+0000", 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0100999999999999e-05 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889861+0000", 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.759e-06 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889865+0000", 2024-04-24T00:34:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6059999999999996e-06 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895025+0000", 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051602059999999996 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895081+0000", 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5625999999999999e-05 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895156+0000", 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4684000000000005e-05 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.897187+0000", 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020309880000000001 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907791+0000", 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010604282 2024-04-24T00:34:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907877+0000", 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6360000000000007e-05 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907943+0000", 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5222000000000007e-05 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.830763001000001, 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01817011, 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 141 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889875+0000", 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2800000000000002e-07 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889879+0000", 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3359999999999999e-06 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889884+0000", 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.31e-06 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.889889+0000", 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8369999999999996e-06 2024-04-24T00:34:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895829+0000", 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059402509999999997 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895895+0000", 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6210999999999993e-05 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.895973+0000", 2024-04-24T00:34:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7540000000000006e-05 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.897198+0000", 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001225414 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.907985+0000", 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010786636 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.908019+0000", 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4258000000000003e-05 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.908045+0000", 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5888999999999999e-05 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:34:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.036743902000001, 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011747629000000001, 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 159 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:34:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683894+0000", 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683895+0000", 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1299999999999998e-07 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683901+0000", 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.823e-06 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683911+0000", 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.004e-05 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.683918+0000", 2024-04-24T00:34:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.294e-06 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.692948+0000", 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090303129999999999 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.693199+0000", 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025027200000000001 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.693589+0000", 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039074299999999998 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695297+0000", 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001707854 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695439+0000", 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014188499999999999 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695582+0000", 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000142473 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:42.695642+0000", 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0019000000000003e-05 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:03.841 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:03.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:03.843 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:03.836+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:03.856 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:34:03.857 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:34:03.873 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-24T00:34:03.651799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6401338,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1417,"num_read_kb":1323,"num_write":468,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":191200,"kb_used_data":2052,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660411680,"statfs":{"total":676457349120,"available":676261560320,"internally_reserved":0,"allocated":2101248,"data_stored":1989758,"data_compressed":12428,"data_compressed_allocated":86016,"data_compressed_original":1232896,"omap_allocated":11157,"internal_metadata":193582187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001012"},"pg_stats":[{"pgid":"3.1f","version":"25'3","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507170+0000","last_change":"2024-04-24T00:33:55.038693+0000","last_active":"2024-04-24T00:33:58.507170+0000","last_peered":"2024-04-24T00:33:58.507170+0000","last_clean":"2024-04-24T00:33:58.507170+0000","last_became_active":"2024-04-24T00:33:53.021533+0000","last_became_peered":"2024-04-24T00:33:53.021533+0000","last_unstale":"2024-04-24T00:33:58.507170+0000","last_undegraded":"2024-04-24T00:33:58.507170+0000","last_fullsized":"2024-04-24T00:33:58.507170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:21.265586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016707930000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443409+0000","last_change":"2024-04-24T00:33:55.023526+0000","last_active":"2024-04-24T00:33:58.443409+0000","last_peered":"2024-04-24T00:33:58.443409+0000","last_clean":"2024-04-24T00:33:58.443409+0000","last_became_active":"2024-04-24T00:33:53.018728+0000","last_became_peered":"2024-04-24T00:33:53.018728+0000","last_unstale":"2024-04-24T00:33:58.443409+0000","last_undegraded":"2024-04-24T00:33:58.443409+0000","last_fullsized":"2024-04-24T00:33:58.443409+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:41:34.466318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026066499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450398+0000","last_change":"2024-04-24T00:33:55.016804+0000","last_active":"2024-04-24T00:33:58.450398+0000","last_peered":"2024-04-24T00:33:58.450398+0000","last_clean":"2024-04-24T00:33:58.450398+0000","last_became_active":"2024-04-24T00:33:53.032813+0000","last_became_peered":"2024-04-24T00:33:53.032813+0000","last_unstale":"2024-04-24T00:33:58.450398+0000","last_undegraded":"2024-04-24T00:33:58.450398+0000","last_fullsized":"2024-04-24T00:33:58.450398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:43:20.528696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085073600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"26'4","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450465+0000","last_change":"2024-04-24T00:33:55.017278+0000","last_active":"2024-04-24T00:33:58.450465+0000","last_peered":"2024-04-24T00:33:58.450465+0000","last_clean":"2024-04-24T00:33:58.450465+0000","last_became_active":"2024-04-24T00:33:53.029450+0000","last_became_peered":"2024-04-24T00:33:53.029450+0000","last_unstale":"2024-04-24T00:33:58.450465+0000","last_undegraded":"2024-04-24T00:33:58.450465+0000","last_fullsized":"2024-04-24T00:33:58.450465+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:30:43.184363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097563100000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"25'10","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507315+0000","last_change":"2024-04-24T00:33:55.040564+0000","last_active":"2024-04-24T00:33:58.507315+0000","last_peered":"2024-04-24T00:33:58.507315+0000","last_clean":"2024-04-24T00:33:58.507315+0000","last_became_active":"2024-04-24T00:33:53.035615+0000","last_became_peered":"2024-04-24T00:33:53.035615+0000","last_unstale":"2024-04-24T00:33:58.507315+0000","last_undegraded":"2024-04-24T00:33:58.507315+0000","last_fullsized":"2024-04-24T00:33:58.507315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:44:17.851552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002442587,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"26'8","reported_seq":75,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.498043+0000","last_change":"2024-04-24T00:33:55.016415+0000","last_active":"2024-04-24T00:33:58.498043+0000","last_peered":"2024-04-24T00:33:58.498043+0000","last_clean":"2024-04-24T00:33:58.498043+0000","last_became_active":"2024-04-24T00:33:53.027055+0000","last_became_peered":"2024-04-24T00:33:53.027055+0000","last_unstale":"2024-04-24T00:33:58.498043+0000","last_undegraded":"2024-04-24T00:33:58.498043+0000","last_fullsized":"2024-04-24T00:33:58.498043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:44.412094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059064600000000001,"stat_sum":{"num_bytes":64,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":43,"num_write":8,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"25'7","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444029+0000","last_change":"2024-04-24T00:33:55.021933+0000","last_active":"2024-04-24T00:33:58.444029+0000","last_peered":"2024-04-24T00:33:58.444029+0000","last_clean":"2024-04-24T00:33:58.444029+0000","last_became_active":"2024-04-24T00:33:53.017357+0000","last_became_peered":"2024-04-24T00:33:53.017357+0000","last_unstale":"2024-04-24T00:33:58.444029+0000","last_undegraded":"2024-04-24T00:33:58.444029+0000","last_fullsized":"2024-04-24T00:33:58.444029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:30:30.203140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019748489999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":26,"num_write":7,"num_write_kb":159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.442937+0000","last_change":"2024-04-24T00:33:55.020096+0000","last_active":"2024-04-24T00:33:58.442937+0000","last_peered":"2024-04-24T00:33:58.442937+0000","last_clean":"2024-04-24T00:33:58.442937+0000","last_became_active":"2024-04-24T00:33:53.019129+0000","last_became_peered":"2024-04-24T00:33:53.019129+0000","last_unstale":"2024-04-24T00:33:58.442937+0000","last_undegraded":"2024-04-24T00:33:58.442937+0000","last_fullsized":"2024-04-24T00:33:58.442937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:25:05.096957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040638499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"25'4","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507205+0000","last_change":"2024-04-24T00:33:55.039924+0000","last_active":"2024-04-24T00:33:58.507205+0000","last_peered":"2024-04-24T00:33:58.507205+0000","last_clean":"2024-04-24T00:33:58.507205+0000","last_became_active":"2024-04-24T00:33:53.018638+0000","last_became_peered":"2024-04-24T00:33:53.018638+0000","last_unstale":"2024-04-24T00:33:58.507205+0000","last_undegraded":"2024-04-24T00:33:58.507205+0000","last_fullsized":"2024-04-24T00:33:58.507205+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:50:28.561834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025650740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":83,"num_write":4,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"26'48","reported_seq":322,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.493329+0000","last_change":"2024-04-24T00:33:55.047327+0000","last_active":"2024-04-24T00:33:58.493329+0000","last_peered":"2024-04-24T00:33:58.493329+0000","last_clean":"2024-04-24T00:33:58.493329+0000","last_became_active":"2024-04-24T00:33:53.025504+0000","last_became_peered":"2024-04-24T00:33:53.025504+0000","last_unstale":"2024-04-24T00:33:58.493329+0000","last_undegraded":"2024-04-24T00:33:58.493329+0000","last_fullsized":"2024-04-24T00:33:58.493329+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:16.074982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025622639999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":707,"num_read_kb":621,"num_write":45,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"25'8","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.446024+0000","last_change":"2024-04-24T00:33:55.045391+0000","last_active":"2024-04-24T00:33:58.446024+0000","last_peered":"2024-04-24T00:33:58.446024+0000","last_clean":"2024-04-24T00:33:58.446024+0000","last_became_active":"2024-04-24T00:33:53.038882+0000","last_became_peered":"2024-04-24T00:33:53.038882+0000","last_unstale":"2024-04-24T00:33:58.446024+0000","last_undegraded":"2024-04-24T00:33:58.446024+0000","last_fullsized":"2024-04-24T00:33:58.446024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:00.435413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067155899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450407+0000","last_change":"2024-04-24T00:33:55.016669+0000","last_active":"2024-04-24T00:33:58.450407+0000","last_peered":"2024-04-24T00:33:58.450407+0000","last_clean":"2024-04-24T00:33:58.450407+0000","last_became_active":"2024-04-24T00:33:53.032030+0000","last_became_peered":"2024-04-24T00:33:53.032030+0000","last_unstale":"2024-04-24T00:33:58.450407+0000","last_undegraded":"2024-04-24T00:33:58.450407+0000","last_fullsized":"2024-04-24T00:33:58.450407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:25.892555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054686100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.454910+0000","last_change":"2024-04-24T00:33:44.961547+0000","last_active":"2024-04-24T00:33:58.454910+0000","last_peered":"2024-04-24T00:33:58.454910+0000","last_clean":"2024-04-24T00:33:58.454910+0000","last_became_active":"2024-04-24T00:33:44.960683+0000","last_became_peered":"2024-04-24T00:33:44.960683+0000","last_unstale":"2024-04-24T00:33:58.454910+0000","last_undegraded":"2024-04-24T00:33:58.454910+0000","last_fullsized":"2024-04-24T00:33:58.454910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507140+0000","last_change":"2024-04-24T00:33:55.038643+0000","last_active":"2024-04-24T00:33:58.507140+0000","last_peered":"2024-04-24T00:33:58.507140+0000","last_clean":"2024-04-24T00:33:58.507140+0000","last_became_active":"2024-04-24T00:33:53.017081+0000","last_became_peered":"2024-04-24T00:33:53.017081+0000","last_unstale":"2024-04-24T00:33:58.507140+0000","last_undegraded":"2024-04-24T00:33:58.507140+0000","last_fullsized":"2024-04-24T00:33:58.507140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:09:17.907662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001587841,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444413+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:33:58.444413+0000","last_peered":"2024-04-24T00:33:58.444413+0000","last_clean":"2024-04-24T00:33:58.444413+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:33:58.444413+0000","last_undegraded":"2024-04-24T00:33:58.444413+0000","last_fullsized":"2024-04-24T00:33:58.444413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443608+0000","last_change":"2024-04-24T00:33:55.022872+0000","last_active":"2024-04-24T00:33:58.443608+0000","last_peered":"2024-04-24T00:33:58.443608+0000","last_clean":"2024-04-24T00:33:58.443608+0000","last_became_active":"2024-04-24T00:33:53.038083+0000","last_became_peered":"2024-04-24T00:33:53.038083+0000","last_unstale":"2024-04-24T00:33:58.443608+0000","last_undegraded":"2024-04-24T00:33:58.443608+0000","last_fullsized":"2024-04-24T00:33:58.443608+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:01.077335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002309391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.446028+0000","last_change":"2024-04-24T00:33:45.648584+0000","last_active":"2024-04-24T00:33:58.446028+0000","last_peered":"2024-04-24T00:33:58.446028+0000","last_clean":"2024-04-24T00:33:58.446028+0000","last_became_active":"2024-04-24T00:33:45.647889+0000","last_became_peered":"2024-04-24T00:33:45.647889+0000","last_unstale":"2024-04-24T00:33:58.446028+0000","last_undegraded":"2024-04-24T00:33:58.446028+0000","last_fullsized":"2024-04-24T00:33:58.446028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:52.085162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"24'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444104+0000","last_change":"2024-04-24T00:33:55.022651+0000","last_active":"2024-04-24T00:33:58.444104+0000","last_peered":"2024-04-24T00:33:58.444104+0000","last_clean":"2024-04-24T00:33:58.444104+0000","last_became_active":"2024-04-24T00:33:53.015167+0000","last_became_peered":"2024-04-24T00:33:53.015167+0000","last_unstale":"2024-04-24T00:33:58.444104+0000","last_undegraded":"2024-04-24T00:33:58.444104+0000","last_fullsized":"2024-04-24T00:33:58.444104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:44.645670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00171313,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444124+0000","last_change":"2024-04-24T00:33:45.650424+0000","last_active":"2024-04-24T00:33:58.444124+0000","last_peered":"2024-04-24T00:33:58.444124+0000","last_clean":"2024-04-24T00:33:58.444124+0000","last_became_active":"2024-04-24T00:33:45.649310+0000","last_became_peered":"2024-04-24T00:33:45.649310+0000","last_unstale":"2024-04-24T00:33:58.444124+0000","last_undegraded":"2024-04-24T00:33:58.444124+0000","last_fullsized":"2024-04-24T00:33:58.444124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:40:40.748700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"25'5","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443362+0000","last_change":"2024-04-24T00:33:55.022033+0000","last_active":"2024-04-24T00:33:58.443362+0000","last_peered":"2024-04-24T00:33:58.443362+0000","last_clean":"2024-04-24T00:33:58.443362+0000","last_became_active":"2024-04-24T00:33:53.016877+0000","last_became_peered":"2024-04-24T00:33:53.016877+0000","last_unstale":"2024-04-24T00:33:58.443362+0000","last_undegraded":"2024-04-24T00:33:58.443362+0000","last_fullsized":"2024-04-24T00:33:58.443362+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:38.617616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001582913,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"24'3","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450383+0000","last_change":"2024-04-24T00:33:55.017280+0000","last_active":"2024-04-24T00:33:58.450383+0000","last_peered":"2024-04-24T00:33:58.450383+0000","last_clean":"2024-04-24T00:33:58.450383+0000","last_became_active":"2024-04-24T00:33:53.031528+0000","last_became_peered":"2024-04-24T00:33:53.031528+0000","last_unstale":"2024-04-24T00:33:58.450383+0000","last_undegraded":"2024-04-24T00:33:58.450383+0000","last_fullsized":"2024-04-24T00:33:58.450383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:06:17.962606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001038394,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444239+0000","last_change":"2024-04-24T00:33:53.019966+0000","last_active":"2024-04-24T00:33:58.444239+0000","last_peered":"2024-04-24T00:33:58.444239+0000","last_clean":"2024-04-24T00:33:58.444239+0000","last_became_active":"2024-04-24T00:33:53.019543+0000","last_became_peered":"2024-04-24T00:33:53.019543+0000","last_unstale":"2024-04-24T00:33:58.444239+0000","last_undegraded":"2024-04-24T00:33:58.444239+0000","last_fullsized":"2024-04-24T00:33:58.444239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:38:43.125607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443974+0000","last_change":"2024-04-24T00:33:55.022161+0000","last_active":"2024-04-24T00:33:58.443974+0000","last_peered":"2024-04-24T00:33:58.443974+0000","last_clean":"2024-04-24T00:33:58.443974+0000","last_became_active":"2024-04-24T00:33:53.012086+0000","last_became_peered":"2024-04-24T00:33:53.012086+0000","last_unstale":"2024-04-24T00:33:58.443974+0000","last_undegraded":"2024-04-24T00:33:58.443974+0000","last_fullsized":"2024-04-24T00:33:58.443974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:43:01.713964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022865120000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.441960+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:33:58.441960+0000","last_peered":"2024-04-24T00:33:58.441960+0000","last_clean":"2024-04-24T00:33:58.441960+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:33:58.441960+0000","last_undegraded":"2024-04-24T00:33:58.441960+0000","last_fullsized":"2024-04-24T00:33:58.441960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"25'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507031+0000","last_change":"2024-04-24T00:33:55.039564+0000","last_active":"2024-04-24T00:33:58.507031+0000","last_peered":"2024-04-24T00:33:58.507031+0000","last_clean":"2024-04-24T00:33:58.507031+0000","last_became_active":"2024-04-24T00:33:53.032926+0000","last_became_peered":"2024-04-24T00:33:53.032926+0000","last_unstale":"2024-04-24T00:33:58.507031+0000","last_undegraded":"2024-04-24T00:33:58.507031+0000","last_fullsized":"2024-04-24T00:33:58.507031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:57:24.226360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023878279999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":26,"num_write":4,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445666+0000","last_change":"2024-04-24T00:33:44.961018+0000","last_active":"2024-04-24T00:33:58.445666+0000","last_peered":"2024-04-24T00:33:58.445666+0000","last_clean":"2024-04-24T00:33:58.445666+0000","last_became_active":"2024-04-24T00:33:44.959946+0000","last_became_peered":"2024-04-24T00:33:44.959946+0000","last_unstale":"2024-04-24T00:33:58.445666+0000","last_undegraded":"2024-04-24T00:33:58.445666+0000","last_fullsized":"2024-04-24T00:33:58.445666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:12.966651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"26'4","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.513987+0000","last_change":"2024-04-24T00:33:55.022861+0000","last_active":"2024-04-24T00:33:58.513987+0000","last_peered":"2024-04-24T00:33:58.513987+0000","last_clean":"2024-04-24T00:33:58.513987+0000","last_became_active":"2024-04-24T00:33:53.019204+0000","last_became_peered":"2024-04-24T00:33:53.019204+0000","last_unstale":"2024-04-24T00:33:58.513987+0000","last_undegraded":"2024-04-24T00:33:58.513987+0000","last_fullsized":"2024-04-24T00:33:58.513987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:26:58.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002279035,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":4,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.442161+0000","last_change":"2024-04-24T00:33:53.018981+0000","last_active":"2024-04-24T00:33:58.442161+0000","last_peered":"2024-04-24T00:33:58.442161+0000","last_clean":"2024-04-24T00:33:58.442161+0000","last_became_active":"2024-04-24T00:33:53.018648+0000","last_became_peered":"2024-04-24T00:33:53.018648+0000","last_unstale":"2024-04-24T00:33:58.442161+0000","last_undegraded":"2024-04-24T00:33:58.442161+0000","last_fullsized":"2024-04-24T00:33:58.442161+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:17.520316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445667+0000","last_change":"2024-04-24T00:33:41.026859+0000","last_active":"2024-04-24T00:33:58.445667+0000","last_peered":"2024-04-24T00:33:58.445667+0000","last_clean":"2024-04-24T00:33:58.445667+0000","last_became_active":"2024-04-24T00:33:41.024668+0000","last_became_peered":"2024-04-24T00:33:41.024668+0000","last_unstale":"2024-04-24T00:33:58.445667+0000","last_undegraded":"2024-04-24T00:33:58.445667+0000","last_fullsized":"2024-04-24T00:33:58.445667+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:25.500752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"24'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443121+0000","last_change":"2024-04-24T00:33:55.020088+0000","last_active":"2024-04-24T00:33:58.443121+0000","last_peered":"2024-04-24T00:33:58.443121+0000","last_clean":"2024-04-24T00:33:58.443121+0000","last_became_active":"2024-04-24T00:33:53.022355+0000","last_became_peered":"2024-04-24T00:33:53.022355+0000","last_unstale":"2024-04-24T00:33:58.443121+0000","last_undegraded":"2024-04-24T00:33:58.443121+0000","last_fullsized":"2024-04-24T00:33:58.443121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:09.977171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053182800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450295+0000","last_change":"2024-04-24T00:33:55.016918+0000","last_active":"2024-04-24T00:33:58.450295+0000","last_peered":"2024-04-24T00:33:58.450295+0000","last_clean":"2024-04-24T00:33:58.450295+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:33:58.450295+0000","last_undegraded":"2024-04-24T00:33:58.450295+0000","last_fullsized":"2024-04-24T00:33:58.450295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:48:16.243483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071856299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.454776+0000","last_change":"2024-04-24T00:33:55.053894+0000","last_active":"2024-04-24T00:33:58.454776+0000","last_peered":"2024-04-24T00:33:58.454776+0000","last_clean":"2024-04-24T00:33:58.454776+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:33:58.454776+0000","last_undegraded":"2024-04-24T00:33:58.454776+0000","last_fullsized":"2024-04-24T00:33:58.454776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:45:07.582887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036442400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"25'10","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443236+0000","last_change":"2024-04-24T00:33:55.023189+0000","last_active":"2024-04-24T00:33:58.443236+0000","last_peered":"2024-04-24T00:33:58.443236+0000","last_clean":"2024-04-24T00:33:58.443236+0000","last_became_active":"2024-04-24T00:33:53.019011+0000","last_became_peered":"2024-04-24T00:33:53.019011+0000","last_unstale":"2024-04-24T00:33:58.443236+0000","last_undegraded":"2024-04-24T00:33:58.443236+0000","last_fullsized":"2024-04-24T00:33:58.443236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:44.886182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025274630000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"26'107","reported_seq":131,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.456200+0000","last_change":"2024-04-24T00:33:55.021383+0000","last_active":"2024-04-24T00:33:58.456200+0000","last_peered":"2024-04-24T00:33:58.456200+0000","last_clean":"2024-04-24T00:33:58.456200+0000","last_became_active":"2024-04-24T00:33:53.024773+0000","last_became_peered":"2024-04-24T00:33:53.024773+0000","last_unstale":"2024-04-24T00:33:58.456200+0000","last_undegraded":"2024-04-24T00:33:58.456200+0000","last_fullsized":"2024-04-24T00:33:58.456200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:27:35.202115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083808799999999996,"stat_sum":{"num_bytes":2786689,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":492,"num_read_kb":390,"num_write":170,"num_write_kb":294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445921+0000","last_change":"2024-04-24T00:33:55.048595+0000","last_active":"2024-04-24T00:33:58.445921+0000","last_peered":"2024-04-24T00:33:58.445921+0000","last_clean":"2024-04-24T00:33:58.445921+0000","last_became_active":"2024-04-24T00:33:53.028960+0000","last_became_peered":"2024-04-24T00:33:53.028960+0000","last_unstale":"2024-04-24T00:33:58.445921+0000","last_undegraded":"2024-04-24T00:33:58.445921+0000","last_fullsized":"2024-04-24T00:33:58.445921+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:58:05.357316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003359524,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"25'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.446002+0000","last_change":"2024-04-24T00:33:55.048272+0000","last_active":"2024-04-24T00:33:58.446002+0000","last_peered":"2024-04-24T00:33:58.446002+0000","last_clean":"2024-04-24T00:33:58.446002+0000","last_became_active":"2024-04-24T00:33:53.031663+0000","last_became_peered":"2024-04-24T00:33:53.031663+0000","last_unstale":"2024-04-24T00:33:58.446002+0000","last_undegraded":"2024-04-24T00:33:58.446002+0000","last_fullsized":"2024-04-24T00:33:58.446002+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:14:36.465508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028975759999999998,"stat_sum":{"num_bytes":3024147,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"24'4","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.446003+0000","last_change":"2024-04-24T00:33:55.047763+0000","last_active":"2024-04-24T00:33:58.446003+0000","last_peered":"2024-04-24T00:33:58.446003+0000","last_clean":"2024-04-24T00:33:58.446003+0000","last_became_active":"2024-04-24T00:33:53.029304+0000","last_became_peered":"2024-04-24T00:33:53.029304+0000","last_unstale":"2024-04-24T00:33:58.446003+0000","last_undegraded":"2024-04-24T00:33:58.446003+0000","last_fullsized":"2024-04-24T00:33:58.446003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:59.933701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024961649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"26'6","reported_seq":40,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.534165+0000","last_change":"2024-04-24T00:33:55.054231+0000","last_active":"2024-04-24T00:33:58.534165+0000","last_peered":"2024-04-24T00:33:58.534165+0000","last_clean":"2024-04-24T00:33:58.534165+0000","last_became_active":"2024-04-24T00:33:53.019812+0000","last_became_peered":"2024-04-24T00:33:53.019812+0000","last_unstale":"2024-04-24T00:33:58.534165+0000","last_undegraded":"2024-04-24T00:33:58.534165+0000","last_fullsized":"2024-04-24T00:33:58.534165+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:03.476602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000661894,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":6,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445817+0000","last_change":"2024-04-24T00:33:55.047919+0000","last_active":"2024-04-24T00:33:58.445817+0000","last_peered":"2024-04-24T00:33:58.445817+0000","last_clean":"2024-04-24T00:33:58.445817+0000","last_became_active":"2024-04-24T00:33:53.026709+0000","last_became_peered":"2024-04-24T00:33:53.026709+0000","last_unstale":"2024-04-24T00:33:58.445817+0000","last_undegraded":"2024-04-24T00:33:58.445817+0000","last_fullsized":"2024-04-24T00:33:58.445817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:48.164766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029753879999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"26'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.534771+0000","last_change":"2024-04-24T00:33:55.040697+0000","last_active":"2024-04-24T00:33:58.534771+0000","last_peered":"2024-04-24T00:33:58.534771+0000","last_clean":"2024-04-24T00:33:58.534771+0000","last_became_active":"2024-04-24T00:33:53.026957+0000","last_became_peered":"2024-04-24T00:33:53.026957+0000","last_unstale":"2024-04-24T00:33:58.534771+0000","last_undegraded":"2024-04-24T00:33:58.534771+0000","last_fullsized":"2024-04-24T00:33:58.534771+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:12:24.020758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003132246,"stat_sum":{"num_bytes":38,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":4,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445900+0000","last_change":"2024-04-24T00:33:55.047334+0000","last_active":"2024-04-24T00:33:58.445900+0000","last_peered":"2024-04-24T00:33:58.445900+0000","last_clean":"2024-04-24T00:33:58.445900+0000","last_became_active":"2024-04-24T00:33:53.033430+0000","last_became_peered":"2024-04-24T00:33:53.033430+0000","last_unstale":"2024-04-24T00:33:58.445900+0000","last_undegraded":"2024-04-24T00:33:58.445900+0000","last_fullsized":"2024-04-24T00:33:58.445900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:26:35.753294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002438646,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5810970,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1301,"num_read_kb":1223,"num_write":309,"num_write_kb":1719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":122160,"data_compressed":234,"data_compressed_allocated":16384,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":64,"acting":64,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":724136,"data_compressed":6187,"data_compressed_allocated":40960,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":129843,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":55834574855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":133787,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":98296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":104414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":104462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":694820,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":64,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":25445,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":35581,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":25445,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":35555,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:34:03.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:34:03.965 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:03.960+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:03.966 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:03.966 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:03.966 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:03.966 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:03.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:04.043 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:04.040+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:04.144 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:04.140+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:04.146 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:04.146 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:04.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:34:04.222 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:34:04.239 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-24T00:34:03.651799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6401338,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1417,"num_read_kb":1323,"num_write":468,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":191200,"kb_used_data":2052,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660411680,"statfs":{"total":676457349120,"available":676261560320,"internally_reserved":0,"allocated":2101248,"data_stored":1989758,"data_compressed":12428,"data_compressed_allocated":86016,"data_compressed_original":1232896,"omap_allocated":11157,"internal_metadata":193582187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001012"},"pg_stats":[{"pgid":"3.1f","version":"25'3","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507170+0000","last_change":"2024-04-24T00:33:55.038693+0000","last_active":"2024-04-24T00:33:58.507170+0000","last_peered":"2024-04-24T00:33:58.507170+0000","last_clean":"2024-04-24T00:33:58.507170+0000","last_became_active":"2024-04-24T00:33:53.021533+0000","last_became_peered":"2024-04-24T00:33:53.021533+0000","last_unstale":"2024-04-24T00:33:58.507170+0000","last_undegraded":"2024-04-24T00:33:58.507170+0000","last_fullsized":"2024-04-24T00:33:58.507170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:21.265586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016707930000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443409+0000","last_change":"2024-04-24T00:33:55.023526+0000","last_active":"2024-04-24T00:33:58.443409+0000","last_peered":"2024-04-24T00:33:58.443409+0000","last_clean":"2024-04-24T00:33:58.443409+0000","last_became_active":"2024-04-24T00:33:53.018728+0000","last_became_peered":"2024-04-24T00:33:53.018728+0000","last_unstale":"2024-04-24T00:33:58.443409+0000","last_undegraded":"2024-04-24T00:33:58.443409+0000","last_fullsized":"2024-04-24T00:33:58.443409+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:41:34.466318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026066499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450398+0000","last_change":"2024-04-24T00:33:55.016804+0000","last_active":"2024-04-24T00:33:58.450398+0000","last_peered":"2024-04-24T00:33:58.450398+0000","last_clean":"2024-04-24T00:33:58.450398+0000","last_became_active":"2024-04-24T00:33:53.032813+0000","last_became_peered":"2024-04-24T00:33:53.032813+0000","last_unstale":"2024-04-24T00:33:58.450398+0000","last_undegraded":"2024-04-24T00:33:58.450398+0000","last_fullsized":"2024-04-24T00:33:58.450398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:43:20.528696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085073600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"26'4","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450465+0000","last_change":"2024-04-24T00:33:55.017278+0000","last_active":"2024-04-24T00:33:58.450465+0000","last_peered":"2024-04-24T00:33:58.450465+0000","last_clean":"2024-04-24T00:33:58.450465+0000","last_became_active":"2024-04-24T00:33:53.029450+0000","last_became_peered":"2024-04-24T00:33:53.029450+0000","last_unstale":"2024-04-24T00:33:58.450465+0000","last_undegraded":"2024-04-24T00:33:58.450465+0000","last_fullsized":"2024-04-24T00:33:58.450465+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:30:43.184363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097563100000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"25'10","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507315+0000","last_change":"2024-04-24T00:33:55.040564+0000","last_active":"2024-04-24T00:33:58.507315+0000","last_peered":"2024-04-24T00:33:58.507315+0000","last_clean":"2024-04-24T00:33:58.507315+0000","last_became_active":"2024-04-24T00:33:53.035615+0000","last_became_peered":"2024-04-24T00:33:53.035615+0000","last_unstale":"2024-04-24T00:33:58.507315+0000","last_undegraded":"2024-04-24T00:33:58.507315+0000","last_fullsized":"2024-04-24T00:33:58.507315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:44:17.851552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002442587,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"26'8","reported_seq":75,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.498043+0000","last_change":"2024-04-24T00:33:55.016415+0000","last_active":"2024-04-24T00:33:58.498043+0000","last_peered":"2024-04-24T00:33:58.498043+0000","last_clean":"2024-04-24T00:33:58.498043+0000","last_became_active":"2024-04-24T00:33:53.027055+0000","last_became_peered":"2024-04-24T00:33:53.027055+0000","last_unstale":"2024-04-24T00:33:58.498043+0000","last_undegraded":"2024-04-24T00:33:58.498043+0000","last_fullsized":"2024-04-24T00:33:58.498043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:44.412094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059064600000000001,"stat_sum":{"num_bytes":64,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":43,"num_write":8,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"25'7","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444029+0000","last_change":"2024-04-24T00:33:55.021933+0000","last_active":"2024-04-24T00:33:58.444029+0000","last_peered":"2024-04-24T00:33:58.444029+0000","last_clean":"2024-04-24T00:33:58.444029+0000","last_became_active":"2024-04-24T00:33:53.017357+0000","last_became_peered":"2024-04-24T00:33:53.017357+0000","last_unstale":"2024-04-24T00:33:58.444029+0000","last_undegraded":"2024-04-24T00:33:58.444029+0000","last_fullsized":"2024-04-24T00:33:58.444029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:30:30.203140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019748489999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":26,"num_write":7,"num_write_kb":159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.442937+0000","last_change":"2024-04-24T00:33:55.020096+0000","last_active":"2024-04-24T00:33:58.442937+0000","last_peered":"2024-04-24T00:33:58.442937+0000","last_clean":"2024-04-24T00:33:58.442937+0000","last_became_active":"2024-04-24T00:33:53.019129+0000","last_became_peered":"2024-04-24T00:33:53.019129+0000","last_unstale":"2024-04-24T00:33:58.442937+0000","last_undegraded":"2024-04-24T00:33:58.442937+0000","last_fullsized":"2024-04-24T00:33:58.442937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:25:05.096957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040638499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"25'4","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507205+0000","last_change":"2024-04-24T00:33:55.039924+0000","last_active":"2024-04-24T00:33:58.507205+0000","last_peered":"2024-04-24T00:33:58.507205+0000","last_clean":"2024-04-24T00:33:58.507205+0000","last_became_active":"2024-04-24T00:33:53.018638+0000","last_became_peered":"2024-04-24T00:33:53.018638+0000","last_unstale":"2024-04-24T00:33:58.507205+0000","last_undegraded":"2024-04-24T00:33:58.507205+0000","last_fullsized":"2024-04-24T00:33:58.507205+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:50:28.561834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025650740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":83,"num_write":4,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"26'48","reported_seq":322,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.493329+0000","last_change":"2024-04-24T00:33:55.047327+0000","last_active":"2024-04-24T00:33:58.493329+0000","last_peered":"2024-04-24T00:33:58.493329+0000","last_clean":"2024-04-24T00:33:58.493329+0000","last_became_active":"2024-04-24T00:33:53.025504+0000","last_became_peered":"2024-04-24T00:33:53.025504+0000","last_unstale":"2024-04-24T00:33:58.493329+0000","last_undegraded":"2024-04-24T00:33:58.493329+0000","last_fullsized":"2024-04-24T00:33:58.493329+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:16.074982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025622639999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":707,"num_read_kb":621,"num_write":45,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"25'8","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.446024+0000","last_change":"2024-04-24T00:33:55.045391+0000","last_active":"2024-04-24T00:33:58.446024+0000","last_peered":"2024-04-24T00:33:58.446024+0000","last_clean":"2024-04-24T00:33:58.446024+0000","last_became_active":"2024-04-24T00:33:53.038882+0000","last_became_peered":"2024-04-24T00:33:53.038882+0000","last_unstale":"2024-04-24T00:33:58.446024+0000","last_undegraded":"2024-04-24T00:33:58.446024+0000","last_fullsized":"2024-04-24T00:33:58.446024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:00.435413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067155899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450407+0000","last_change":"2024-04-24T00:33:55.016669+0000","last_active":"2024-04-24T00:33:58.450407+0000","last_peered":"2024-04-24T00:33:58.450407+0000","last_clean":"2024-04-24T00:33:58.450407+0000","last_became_active":"2024-04-24T00:33:53.032030+0000","last_became_peered":"2024-04-24T00:33:53.032030+0000","last_unstale":"2024-04-24T00:33:58.450407+0000","last_undegraded":"2024-04-24T00:33:58.450407+0000","last_fullsized":"2024-04-24T00:33:58.450407+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:25.892555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054686100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.454910+0000","last_change":"2024-04-24T00:33:44.961547+0000","last_active":"2024-04-24T00:33:58.454910+0000","last_peered":"2024-04-24T00:33:58.454910+0000","last_clean":"2024-04-24T00:33:58.454910+0000","last_became_active":"2024-04-24T00:33:44.960683+0000","last_became_peered":"2024-04-24T00:33:44.960683+0000","last_unstale":"2024-04-24T00:33:58.454910+0000","last_undegraded":"2024-04-24T00:33:58.454910+0000","last_fullsized":"2024-04-24T00:33:58.454910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507140+0000","last_change":"2024-04-24T00:33:55.038643+0000","last_active":"2024-04-24T00:33:58.507140+0000","last_peered":"2024-04-24T00:33:58.507140+0000","last_clean":"2024-04-24T00:33:58.507140+0000","last_became_active":"2024-04-24T00:33:53.017081+0000","last_became_peered":"2024-04-24T00:33:53.017081+0000","last_unstale":"2024-04-24T00:33:58.507140+0000","last_undegraded":"2024-04-24T00:33:58.507140+0000","last_fullsized":"2024-04-24T00:33:58.507140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:09:17.907662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001587841,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444413+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:33:58.444413+0000","last_peered":"2024-04-24T00:33:58.444413+0000","last_clean":"2024-04-24T00:33:58.444413+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:33:58.444413+0000","last_undegraded":"2024-04-24T00:33:58.444413+0000","last_fullsized":"2024-04-24T00:33:58.444413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443608+0000","last_change":"2024-04-24T00:33:55.022872+0000","last_active":"2024-04-24T00:33:58.443608+0000","last_peered":"2024-04-24T00:33:58.443608+0000","last_clean":"2024-04-24T00:33:58.443608+0000","last_became_active":"2024-04-24T00:33:53.038083+0000","last_became_peered":"2024-04-24T00:33:53.038083+0000","last_unstale":"2024-04-24T00:33:58.443608+0000","last_undegraded":"2024-04-24T00:33:58.443608+0000","last_fullsized":"2024-04-24T00:33:58.443608+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:01:01.077335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002309391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.446028+0000","last_change":"2024-04-24T00:33:45.648584+0000","last_active":"2024-04-24T00:33:58.446028+0000","last_peered":"2024-04-24T00:33:58.446028+0000","last_clean":"2024-04-24T00:33:58.446028+0000","last_became_active":"2024-04-24T00:33:45.647889+0000","last_became_peered":"2024-04-24T00:33:45.647889+0000","last_unstale":"2024-04-24T00:33:58.446028+0000","last_undegraded":"2024-04-24T00:33:58.446028+0000","last_fullsized":"2024-04-24T00:33:58.446028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:52.085162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"24'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444104+0000","last_change":"2024-04-24T00:33:55.022651+0000","last_active":"2024-04-24T00:33:58.444104+0000","last_peered":"2024-04-24T00:33:58.444104+0000","last_clean":"2024-04-24T00:33:58.444104+0000","last_became_active":"2024-04-24T00:33:53.015167+0000","last_became_peered":"2024-04-24T00:33:53.015167+0000","last_unstale":"2024-04-24T00:33:58.444104+0000","last_undegraded":"2024-04-24T00:33:58.444104+0000","last_fullsized":"2024-04-24T00:33:58.444104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:44.645670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00171313,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444124+0000","last_change":"2024-04-24T00:33:45.650424+0000","last_active":"2024-04-24T00:33:58.444124+0000","last_peered":"2024-04-24T00:33:58.444124+0000","last_clean":"2024-04-24T00:33:58.444124+0000","last_became_active":"2024-04-24T00:33:45.649310+0000","last_became_peered":"2024-04-24T00:33:45.649310+0000","last_unstale":"2024-04-24T00:33:58.444124+0000","last_undegraded":"2024-04-24T00:33:58.444124+0000","last_fullsized":"2024-04-24T00:33:58.444124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:40:40.748700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"25'5","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443362+0000","last_change":"2024-04-24T00:33:55.022033+0000","last_active":"2024-04-24T00:33:58.443362+0000","last_peered":"2024-04-24T00:33:58.443362+0000","last_clean":"2024-04-24T00:33:58.443362+0000","last_became_active":"2024-04-24T00:33:53.016877+0000","last_became_peered":"2024-04-24T00:33:53.016877+0000","last_unstale":"2024-04-24T00:33:58.443362+0000","last_undegraded":"2024-04-24T00:33:58.443362+0000","last_fullsized":"2024-04-24T00:33:58.443362+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:38.617616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001582913,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"24'3","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450383+0000","last_change":"2024-04-24T00:33:55.017280+0000","last_active":"2024-04-24T00:33:58.450383+0000","last_peered":"2024-04-24T00:33:58.450383+0000","last_clean":"2024-04-24T00:33:58.450383+0000","last_became_active":"2024-04-24T00:33:53.031528+0000","last_became_peered":"2024-04-24T00:33:53.031528+0000","last_unstale":"2024-04-24T00:33:58.450383+0000","last_undegraded":"2024-04-24T00:33:58.450383+0000","last_fullsized":"2024-04-24T00:33:58.450383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:06:17.962606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001038394,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.444239+0000","last_change":"2024-04-24T00:33:53.019966+0000","last_active":"2024-04-24T00:33:58.444239+0000","last_peered":"2024-04-24T00:33:58.444239+0000","last_clean":"2024-04-24T00:33:58.444239+0000","last_became_active":"2024-04-24T00:33:53.019543+0000","last_became_peered":"2024-04-24T00:33:53.019543+0000","last_unstale":"2024-04-24T00:33:58.444239+0000","last_undegraded":"2024-04-24T00:33:58.444239+0000","last_fullsized":"2024-04-24T00:33:58.444239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:38:43.125607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443974+0000","last_change":"2024-04-24T00:33:55.022161+0000","last_active":"2024-04-24T00:33:58.443974+0000","last_peered":"2024-04-24T00:33:58.443974+0000","last_clean":"2024-04-24T00:33:58.443974+0000","last_became_active":"2024-04-24T00:33:53.012086+0000","last_became_peered":"2024-04-24T00:33:53.012086+0000","last_unstale":"2024-04-24T00:33:58.443974+0000","last_undegraded":"2024-04-24T00:33:58.443974+0000","last_fullsized":"2024-04-24T00:33:58.443974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:43:01.713964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022865120000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.441960+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:33:58.441960+0000","last_peered":"2024-04-24T00:33:58.441960+0000","last_clean":"2024-04-24T00:33:58.441960+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:33:58.441960+0000","last_undegraded":"2024-04-24T00:33:58.441960+0000","last_fullsized":"2024-04-24T00:33:58.441960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"25'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.507031+0000","last_change":"2024-04-24T00:33:55.039564+0000","last_active":"2024-04-24T00:33:58.507031+0000","last_peered":"2024-04-24T00:33:58.507031+0000","last_clean":"2024-04-24T00:33:58.507031+0000","last_became_active":"2024-04-24T00:33:53.032926+0000","last_became_peered":"2024-04-24T00:33:53.032926+0000","last_unstale":"2024-04-24T00:33:58.507031+0000","last_undegraded":"2024-04-24T00:33:58.507031+0000","last_fullsized":"2024-04-24T00:33:58.507031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:57:24.226360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023878279999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":26,"num_write":4,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445666+0000","last_change":"2024-04-24T00:33:44.961018+0000","last_active":"2024-04-24T00:33:58.445666+0000","last_peered":"2024-04-24T00:33:58.445666+0000","last_clean":"2024-04-24T00:33:58.445666+0000","last_became_active":"2024-04-24T00:33:44.959946+0000","last_became_peered":"2024-04-24T00:33:44.959946+0000","last_unstale":"2024-04-24T00:33:58.445666+0000","last_undegraded":"2024-04-24T00:33:58.445666+0000","last_fullsized":"2024-04-24T00:33:58.445666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:12.966651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"26'4","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.513987+0000","last_change":"2024-04-24T00:33:55.022861+0000","last_active":"2024-04-24T00:33:58.513987+0000","last_peered":"2024-04-24T00:33:58.513987+0000","last_clean":"2024-04-24T00:33:58.513987+0000","last_became_active":"2024-04-24T00:33:53.019204+0000","last_became_peered":"2024-04-24T00:33:53.019204+0000","last_unstale":"2024-04-24T00:33:58.513987+0000","last_undegraded":"2024-04-24T00:33:58.513987+0000","last_fullsized":"2024-04-24T00:33:58.513987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:26:58.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002279035,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":4,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.442161+0000","last_change":"2024-04-24T00:33:53.018981+0000","last_active":"2024-04-24T00:33:58.442161+0000","last_peered":"2024-04-24T00:33:58.442161+0000","last_clean":"2024-04-24T00:33:58.442161+0000","last_became_active":"2024-04-24T00:33:53.018648+0000","last_became_peered":"2024-04-24T00:33:53.018648+0000","last_unstale":"2024-04-24T00:33:58.442161+0000","last_undegraded":"2024-04-24T00:33:58.442161+0000","last_fullsized":"2024-04-24T00:33:58.442161+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:17.520316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445667+0000","last_change":"2024-04-24T00:33:41.026859+0000","last_active":"2024-04-24T00:33:58.445667+0000","last_peered":"2024-04-24T00:33:58.445667+0000","last_clean":"2024-04-24T00:33:58.445667+0000","last_became_active":"2024-04-24T00:33:41.024668+0000","last_became_peered":"2024-04-24T00:33:41.024668+0000","last_unstale":"2024-04-24T00:33:58.445667+0000","last_undegraded":"2024-04-24T00:33:58.445667+0000","last_fullsized":"2024-04-24T00:33:58.445667+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:25.500752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"24'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443121+0000","last_change":"2024-04-24T00:33:55.020088+0000","last_active":"2024-04-24T00:33:58.443121+0000","last_peered":"2024-04-24T00:33:58.443121+0000","last_clean":"2024-04-24T00:33:58.443121+0000","last_became_active":"2024-04-24T00:33:53.022355+0000","last_became_peered":"2024-04-24T00:33:53.022355+0000","last_unstale":"2024-04-24T00:33:58.443121+0000","last_undegraded":"2024-04-24T00:33:58.443121+0000","last_fullsized":"2024-04-24T00:33:58.443121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:09.977171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053182800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.450295+0000","last_change":"2024-04-24T00:33:55.016918+0000","last_active":"2024-04-24T00:33:58.450295+0000","last_peered":"2024-04-24T00:33:58.450295+0000","last_clean":"2024-04-24T00:33:58.450295+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:33:58.450295+0000","last_undegraded":"2024-04-24T00:33:58.450295+0000","last_fullsized":"2024-04-24T00:33:58.450295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:48:16.243483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071856299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.454776+0000","last_change":"2024-04-24T00:33:55.053894+0000","last_active":"2024-04-24T00:33:58.454776+0000","last_peered":"2024-04-24T00:33:58.454776+0000","last_clean":"2024-04-24T00:33:58.454776+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:33:58.454776+0000","last_undegraded":"2024-04-24T00:33:58.454776+0000","last_fullsized":"2024-04-24T00:33:58.454776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:45:07.582887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036442400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"25'10","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.443236+0000","last_change":"2024-04-24T00:33:55.023189+0000","last_active":"2024-04-24T00:33:58.443236+0000","last_peered":"2024-04-24T00:33:58.443236+0000","last_clean":"2024-04-24T00:33:58.443236+0000","last_became_active":"2024-04-24T00:33:53.019011+0000","last_became_peered":"2024-04-24T00:33:53.019011+0000","last_unstale":"2024-04-24T00:33:58.443236+0000","last_undegraded":"2024-04-24T00:33:58.443236+0000","last_fullsized":"2024-04-24T00:33:58.443236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:44.886182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025274630000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"26'107","reported_seq":131,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.456200+0000","last_change":"2024-04-24T00:33:55.021383+0000","last_active":"2024-04-24T00:33:58.456200+0000","last_peered":"2024-04-24T00:33:58.456200+0000","last_clean":"2024-04-24T00:33:58.456200+0000","last_became_active":"2024-04-24T00:33:53.024773+0000","last_became_peered":"2024-04-24T00:33:53.024773+0000","last_unstale":"2024-04-24T00:33:58.456200+0000","last_undegraded":"2024-04-24T00:33:58.456200+0000","last_fullsized":"2024-04-24T00:33:58.456200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:27:35.202115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083808799999999996,"stat_sum":{"num_bytes":2786689,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":492,"num_read_kb":390,"num_write":170,"num_write_kb":294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445921+0000","last_change":"2024-04-24T00:33:55.048595+0000","last_active":"2024-04-24T00:33:58.445921+0000","last_peered":"2024-04-24T00:33:58.445921+0000","last_clean":"2024-04-24T00:33:58.445921+0000","last_became_active":"2024-04-24T00:33:53.028960+0000","last_became_peered":"2024-04-24T00:33:53.028960+0000","last_unstale":"2024-04-24T00:33:58.445921+0000","last_undegraded":"2024-04-24T00:33:58.445921+0000","last_fullsized":"2024-04-24T00:33:58.445921+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:58:05.357316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003359524,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"25'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.446002+0000","last_change":"2024-04-24T00:33:55.048272+0000","last_active":"2024-04-24T00:33:58.446002+0000","last_peered":"2024-04-24T00:33:58.446002+0000","last_clean":"2024-04-24T00:33:58.446002+0000","last_became_active":"2024-04-24T00:33:53.031663+0000","last_became_peered":"2024-04-24T00:33:53.031663+0000","last_unstale":"2024-04-24T00:33:58.446002+0000","last_undegraded":"2024-04-24T00:33:58.446002+0000","last_fullsized":"2024-04-24T00:33:58.446002+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:14:36.465508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028975759999999998,"stat_sum":{"num_bytes":3024147,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"24'4","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.446003+0000","last_change":"2024-04-24T00:33:55.047763+0000","last_active":"2024-04-24T00:33:58.446003+0000","last_peered":"2024-04-24T00:33:58.446003+0000","last_clean":"2024-04-24T00:33:58.446003+0000","last_became_active":"2024-04-24T00:33:53.029304+0000","last_became_peered":"2024-04-24T00:33:53.029304+0000","last_unstale":"2024-04-24T00:33:58.446003+0000","last_undegraded":"2024-04-24T00:33:58.446003+0000","last_fullsized":"2024-04-24T00:33:58.446003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:59.933701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024961649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"26'6","reported_seq":40,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.534165+0000","last_change":"2024-04-24T00:33:55.054231+0000","last_active":"2024-04-24T00:33:58.534165+0000","last_peered":"2024-04-24T00:33:58.534165+0000","last_clean":"2024-04-24T00:33:58.534165+0000","last_became_active":"2024-04-24T00:33:53.019812+0000","last_became_peered":"2024-04-24T00:33:53.019812+0000","last_unstale":"2024-04-24T00:33:58.534165+0000","last_undegraded":"2024-04-24T00:33:58.534165+0000","last_fullsized":"2024-04-24T00:33:58.534165+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:03.476602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000661894,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":6,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445817+0000","last_change":"2024-04-24T00:33:55.047919+0000","last_active":"2024-04-24T00:33:58.445817+0000","last_peered":"2024-04-24T00:33:58.445817+0000","last_clean":"2024-04-24T00:33:58.445817+0000","last_became_active":"2024-04-24T00:33:53.026709+0000","last_became_peered":"2024-04-24T00:33:53.026709+0000","last_unstale":"2024-04-24T00:33:58.445817+0000","last_undegraded":"2024-04-24T00:33:58.445817+0000","last_fullsized":"2024-04-24T00:33:58.445817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:48.164766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029753879999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"26'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.534771+0000","last_change":"2024-04-24T00:33:55.040697+0000","last_active":"2024-04-24T00:33:58.534771+0000","last_peered":"2024-04-24T00:33:58.534771+0000","last_clean":"2024-04-24T00:33:58.534771+0000","last_became_active":"2024-04-24T00:33:53.026957+0000","last_became_peered":"2024-04-24T00:33:53.026957+0000","last_unstale":"2024-04-24T00:33:58.534771+0000","last_undegraded":"2024-04-24T00:33:58.534771+0000","last_fullsized":"2024-04-24T00:33:58.534771+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:12:24.020758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003132246,"stat_sum":{"num_bytes":38,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":4,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T00:33:58.445900+0000","last_change":"2024-04-24T00:33:55.047334+0000","last_active":"2024-04-24T00:33:58.445900+0000","last_peered":"2024-04-24T00:33:58.445900+0000","last_clean":"2024-04-24T00:33:58.445900+0000","last_became_active":"2024-04-24T00:33:53.033430+0000","last_became_peered":"2024-04-24T00:33:53.033430+0000","last_unstale":"2024-04-24T00:33:58.445900+0000","last_undegraded":"2024-04-24T00:33:58.445900+0000","last_fullsized":"2024-04-24T00:33:58.445900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:26:35.753294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002438646,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5810970,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1301,"num_read_kb":1223,"num_write":309,"num_write_kb":1719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":122160,"data_compressed":234,"data_compressed_allocated":16384,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":64,"acting":64,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":724136,"data_compressed":6187,"data_compressed_allocated":40960,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":129843,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":55834574855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":133787,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":98296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":104414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":104462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":694820,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":64,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":25445,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":35581,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":25445,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":35555,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:34:04.242 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T00:34:04.242 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:04.242 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T00:34:04.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T00:34:04.245 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:04.240+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:04.283 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:129 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'87, mlcod=16'85)", 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.389709838000002, 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078162660000000005, 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:34:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891115+0000", 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891116+0000", 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.747e-06 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891132+0000", 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5282999999999999e-05 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891161+0000", 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9519e-05 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891175+0000", 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891315+0000", 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140032 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891354+0000", 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9199999999999997e-05 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898842+0000", 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074878690000000003 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898931+0000", 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8988999999999998e-05 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:130 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'88, mlcod=16'85)", 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.38902079, 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098350269999999997, 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891804+0000", 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891805+0000", 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.73e-06 2024-04-24T00:34:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891819+0000", 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.382e-05 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891847+0000", 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8155999999999999e-05 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.891867+0000", 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9919999999999999e-05 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892020+0000", 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152139 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892101+0000", 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1738999999999998e-05 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901528+0000", 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094262470000000004 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901639+0000", 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111276 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:131 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:34:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.388512903999999, 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094484630000000007, 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892312+0000", 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892313+0000", 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7770000000000001e-06 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892385+0000", 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1939000000000004e-05 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892434+0000", 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9023000000000001e-05 2024-04-24T00:34:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892449+0000", 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4263e-05 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.892995+0000", 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054614900000000001 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893032+0000", 2024-04-24T00:34:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6863999999999998e-05 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901646+0000", 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086146460000000001 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901760+0000", 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011380200000000001 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:132 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.387687553999999, 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0086804629999999994, 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893137+0000", 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893139+0000", 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.747e-06 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893152+0000", 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.347e-05 2024-04-24T00:34:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893182+0000", 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9366e-05 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893198+0000", 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6677e-05 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894622+0000", 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014240489999999999 2024-04-24T00:34:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894661+0000", 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9019e-05 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901769+0000", 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071080149999999996 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901817+0000", 2024-04-24T00:34:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.812e-05 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:133 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'85)", 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.387348243000002, 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01116346, 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893476+0000", 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893478+0000", 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3659999999999999e-06 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893490+0000", 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2784e-05 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893519+0000", 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8983e-05 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893544+0000", 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4813000000000001e-05 2024-04-24T00:34:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895382+0000", 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018376759999999999 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895433+0000", 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1093e-05 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904576+0000", 2024-04-24T00:34:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091431289999999998 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904640+0000", 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3615999999999995e-05 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:134 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'85)", 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.386928173000001, 2024-04-24T00:34:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010781628999999999, 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893896+0000", 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893898+0000", 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4670000000000001e-06 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.893990+0000", 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2221999999999999e-05 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894021+0000", 2024-04-24T00:34:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0710000000000002e-05 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894040+0000", 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9303000000000001e-05 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896087+0000", 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020471909999999999 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896120+0000", 2024-04-24T00:34:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3056999999999999e-05 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904643+0000", 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085222530000000005 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904678+0000", 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5426000000000003e-05 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:135 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'85)", 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.386125805999999, 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010018802, 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894699+0000", 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894701+0000", 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.79e-06 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894713+0000", 2024-04-24T00:34:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2737e-05 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894743+0000", 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0086e-05 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.894764+0000", 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1063000000000001e-05 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897677+0000", 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029129609999999999 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897711+0000", 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3763000000000002e-05 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904682+0000", 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069707090000000003 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904718+0000", 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5692999999999999e-05 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:136 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'85)", 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.385744442, 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096742749999999995, 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895080+0000", 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895082+0000", 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.64e-06 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895093+0000", 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895122+0000", 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8989999999999999e-05 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895162+0000", 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9573000000000002e-05 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898343+0000", 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031815279999999999 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898376+0000", 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2610000000000001e-05 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904722+0000", 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006346338 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904754+0000", 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2033000000000003e-05 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:137 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'85)", 2024-04-24T00:34:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.385309732, 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092832910000000008, 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895515+0000", 2024-04-24T00:34:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895517+0000", 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895604+0000", 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6876e-05 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895632+0000", 2024-04-24T00:34:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.834e-05 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.895649+0000", 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7266e-05 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.899032+0000", 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033826339999999998 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.899066+0000", 2024-04-24T00:34:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3603000000000003e-05 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904756+0000", 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056903190000000001 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.904798+0000", 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2132999999999998e-05 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:138 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'96, mlcod=16'85)", 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.384519672, 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010420479, 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896305+0000", 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896307+0000", 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7099999999999999e-06 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896321+0000", 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4847e-05 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896348+0000", 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6216000000000001e-05 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896367+0000", 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.965e-05 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.900665+0000", 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042981570000000004 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.900727+0000", 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1682999999999999e-05 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906588+0000", 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058611999999999996 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906725+0000", 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137016 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:139 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'97, mlcod=16'85)", 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.384175591000002, 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010172130999999999, 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:34:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896649+0000", 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896650+0000", 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.457e-06 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896665+0000", 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5085999999999999e-05 2024-04-24T00:34:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896696+0000", 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0505999999999999e-05 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.896713+0000", 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7139999999999999e-05 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901935+0000", 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052216169999999996 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.901974+0000", 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9632000000000003e-05 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906732+0000", 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047579370000000003 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.906821+0000", 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8756000000000002e-05 2024-04-24T00:34:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:140 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'85)", 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.383676325, 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012183782000000001, 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897149+0000", 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.147e-06 2024-04-24T00:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897223+0000", 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4116000000000007e-05 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897251+0000", 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7606e-05 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897269+0000", 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7476999999999999e-05 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902553+0000", 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052847359999999999 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902586+0000", 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2802000000000002e-05 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909235+0000", 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066493519999999999 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909332+0000", 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6545999999999993e-05 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'85)", 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.382866215, 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011427935, 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897960+0000", 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1500000000000002e-06 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897974+0000", 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3890000000000001e-05 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898000+0000", 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.599e-05 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898018+0000", 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7903000000000001e-05 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903913+0000", 2024-04-24T00:34:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058942839999999996 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903976+0000", 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3847e-05 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909338+0000", 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053611850000000001 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909386+0000", 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8686000000000001e-05 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.248406546, 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097809650000000008, 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 11 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032418+0000", 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032421+0000", 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3730000000000001e-06 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032435+0000", 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3468999999999999e-05 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032461+0000", 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032473+0000", 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1613999999999999e-05 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032624+0000", 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015161999999999999 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.032631+0000", 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8700000000000003e-06 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037449+0000", 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048180180000000003 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037551+0000", 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010177 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037724+0000", 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172943 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039447+0000", 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017229560000000001 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042096+0000", 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002649069 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042164+0000", 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7931999999999999e-05 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.042199+0000", 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123000000000002e-05 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:771 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.480762+0000", 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8000621670000001, 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055778920000000001, 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 771 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480762+0000", 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480762+0000", 2024-04-24T00:34:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480764+0000", 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-06 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480771+0000", 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7700000000000004e-06 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480787+0000", 2024-04-24T00:34:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6739e-05 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.480797+0000", 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7559999999999994e-06 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481743+0000", 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094565100000000004 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481804+0000", 2024-04-24T00:34:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1775999999999999e-05 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.482027+0000", 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022297199999999999 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485225+0000", 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031973269999999998 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486217+0000", 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099223400000000004 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486286+0000", 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8526000000000001e-05 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486340+0000", 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4725999999999997e-05 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:772 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.481122+0000", 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7997029979999999, 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053661680000000002, 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 772 2024-04-24T00:34:04.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481122+0000", 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481122+0000", 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481123+0000", 2024-04-24T00:34:04.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3999999999999999e-06 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481130+0000", 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6800000000000004e-06 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481145+0000", 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5630000000000001e-05 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481155+0000", 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0051000000000001e-05 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.482486+0000", 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001330323 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.482565+0000", 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8925999999999996e-05 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.482973+0000", 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040814099999999999 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485238+0000", 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022650420000000001 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486389+0000", 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001151074 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486444+0000", 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4913000000000001e-05 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486488+0000", 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3988e-05 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:773 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:04.356 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.481193+0000", 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7996313439999998, 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074095879999999999, 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 773 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481193+0000", 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481193+0000", 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481194+0000", 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481200+0000", 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5570000000000003e-06 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481212+0000", 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1911e-05 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481222+0000", 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0662000000000001e-05 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483417+0000", 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002195135 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483490+0000", 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2521000000000006e-05 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483807+0000", 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031740300000000001 2024-04-24T00:34:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486550+0000", 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027422280000000002 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.488483+0000", 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001932991 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.488543+0000", 2024-04-24T00:34:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0028999999999997e-05 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.488603+0000", 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0251000000000003e-05 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:774 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.481503+0000", 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7993216480000003, 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0096041609999999999, 2024-04-24T00:34:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 774 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481503+0000", 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481503+0000", 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481504+0000", 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3340000000000001e-06 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481510+0000", 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2659999999999998e-06 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481525+0000", 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4858000000000001e-05 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.481535+0000", 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2590000000000006e-06 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.484259+0000", 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027241840000000002 2024-04-24T00:34:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.484330+0000", 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1196000000000001e-05 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.484632+0000", 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030203599999999999 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486575+0000", 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019433569999999999 2024-04-24T00:34:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491056+0000", 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004480436 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491084+0000", 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8265000000000001e-05 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491107+0000", 2024-04-24T00:34:04.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2969999999999999e-05 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:775 3.1 3:8747de5b:::rbd_object_map.115eb9c76af6.0000000000000004:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:59.483006+0000", 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7978182540000001, 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084191790000000006, 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 775 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483006+0000", 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483006+0000", 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483007+0000", 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5099999999999998e-07 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483012+0000", 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1100000000000002e-06 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483023+0000", 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0807e-05 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.483046+0000", 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2974000000000001e-05 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485274+0000", 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002227499 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485369+0000", 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5424e-05 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.485757+0000", 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038762099999999999 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.486926+0000", 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011698780000000001 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491211+0000", 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042842310000000003 2024-04-24T00:34:04.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491311+0000", 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:59.491425+0000", 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114499 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:830 3.12 3:4e2177ac:::rbd_object_map.115eb9c76af6:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:00.471740+0000", 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.809084666, 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060239789999999996, 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 830 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.471740+0000", 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.471740+0000", 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.471742+0000", 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4269999999999998e-06 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.471752+0000", 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8949999999999993e-06 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.471769+0000", 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6546999999999999e-05 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.471782+0000", 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3713000000000001e-05 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.471862+0000", 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9134999999999994e-05 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.472010+0000", 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014797200000000001 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 7", 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.472252+0000", 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024198500000000001 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.474628+0000", 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023759699999999998 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.477552+0000", 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002924509 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.477690+0000", 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013803900000000001 2024-04-24T00:34:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:00.477764+0000", 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3787000000000004e-05 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.379 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:04.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:04.381 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:04.376+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:04.482 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:04.476+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:04.497 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:34:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:34:04.541 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 33, 2024-04-24T00:34:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-04-24T00:33:24.857711+0000", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-04-24T00:34:04.523622+0000", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-04-24T00:33:37.880801+0000", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-04-24T00:33:51.992273+0000", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5801984, 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-04-24T00:34:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 4, 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 3, 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "jewel", 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:39.734814+0000", 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-04-24T00:34:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "17", 2024-04-24T00:34:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-04-24T00:34:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 8, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 8, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.25, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-04-24T00:34:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:43.786311+0000", 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "20", 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:34:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.25, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.25, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "pool_client.0", 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:51.288702+0000", 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "32", 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 6, 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 32, 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.9900000095367432, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.9900000095367432, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1c80b3f4-99ee-4725-90a8-920116161bf5", 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 21, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-04-24T00:34:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6817", 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6818", 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6819", 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6822", 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6823", 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6820", 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6821", 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6817/3307242121", 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6819/3307242121", 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6823/3307242121", 2024-04-24T00:34:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6821/3307242121", 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "20e1331a-f8a2-48fd-b2b8-b7a92f21d78e", 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 21, 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6824", 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6825", 2024-04-24T00:34:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6826", 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6827", 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6830", 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6831", 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6828", 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6829", 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6825/637720411", 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6827/637720411", 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6831/637720411", 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6829/637720411", 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:34:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a51557a0-9634-4a20-82b0-b54cdd36b166", 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 21, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/820024667", 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6811/820024667", 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6815/820024667", 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6813/820024667", 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "aafffa79-f532-4441-b1d6-f6441c5eebf8", 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 21, 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-24T00:34:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-04-24T00:34:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-04-24T00:34:04.581 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.582 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/1974534323", 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6803/1974534323", 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1974534323", 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1974534323", 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:04.583 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "78f05556-598e-4be2-b682-e3f58863eb82", 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 21, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:04.584 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6816", 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6818", 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:04.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6820", 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6821", 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6826", 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6828", 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6823", 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6824", 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6818/1642543604", 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6821/1642543604", 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6828/1642543604", 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6824/1642543604", 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:04.588 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5", 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 18, 2024-04-24T00:34:04.589 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6808", 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6809", 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6810", 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6811", 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.591 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6814", 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6815", 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:04.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6812", 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6813", 2024-04-24T00:34:04.593 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6809/172726194", 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6811/172726194", 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6815/172726194", 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6813/172726194", 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "6dad12bf-94de-482a-a402-49e67d5e5dc7", 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 21, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:04.595 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6800", 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6801", 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:04.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6802", 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6803", 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:04.597 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6806", 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6807", 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.598 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6804", 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6805", 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6801/2195746987", 2024-04-24T00:34:04.599 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6803/2195746987", 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2195746987", 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6805/2195746987", 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e6734cfe-3c24-4ddd-8ef0-5ed450efe772", 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:04.600 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 21, 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6817", 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6819", 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6822", 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6825", 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6830", 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6831", 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6827", 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:04.604 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6829", 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6819/954791674", 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6825/954791674", 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6831/954791674", 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6829/954791674", 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.605 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.509233+0000", 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:34:04.606 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.465696+0000", 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:04.607 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.405177+0000", 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.269202+0000", 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.608 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.359180+0000", 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:34:04.609 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.351834+0000", 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.490127+0000", 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.484842+0000", 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [], 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [ 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 4, 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [ 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 2, 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-04-24T00:34:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-04-24T00:34:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-04-24T00:34:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-04-24T00:34:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.615 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:04.617 INFO:tasks.thrashosds.thrasher:j is {'epoch': 33, 'fsid': '6a70d87d-4453-47be-af2a-5590f9851254', 'created': '2024-04-24T00:33:24.857711+0000', 'modified': '2024-04-24T00:34:04.523622+0000', 'last_up_change': '2024-04-24T00:33:37.880801+0000', 'last_in_change': '2024-04-24T00:33:51.992273+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T00:33:39.734814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T00:33:43.786311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-24T00:33:51.288702+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 32, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '1c80b3f4-99ee-4725-90a8-920116161bf5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 3307242121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 3307242121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 3307242121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 3307242121}]}, 'public_addr': '172.21.15.64:6817/3307242121', 'cluster_addr': '172.21.15.64:6819/3307242121', 'heartbeat_back_addr': '172.21.15.64:6823/3307242121', 'heartbeat_front_addr': '172.21.15.64:6821/3307242121', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 637720411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 637720411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 637720411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 637720411}]}, 'public_addr': '172.21.15.64:6825/637720411', 'cluster_addr': '172.21.15.64:6827/637720411', 'heartbeat_back_addr': '172.21.15.64:6831/637720411', 'heartbeat_front_addr': '172.21.15.64:6829/637720411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a51557a0-9634-4a20-82b0-b54cdd36b166', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 820024667}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 820024667}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 820024667}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 820024667}]}, 'public_addr': '172.21.15.64:6809/820024667', 'cluster_addr': '172.21.15.64:6811/820024667', 'heartbeat_back_addr': '172.21.15.64:6815/820024667', 'heartbeat_front_addr': '172.21.15.64:6813/820024667', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aafffa79-f532-4441-b1d6-f6441c5eebf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1974534323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1974534323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1974534323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1974534323}]}, 'public_addr': '172.21.15.64:6801/1974534323', 'cluster_addr': '172.21.15.64:6803/1974534323', 'heartbeat_back_addr': '172.21.15.64:6807/1974534323', 'heartbeat_front_addr': '172.21.15.64:6805/1974534323', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '78f05556-598e-4be2-b682-e3f58863eb82', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6816', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6818', 'nonce': 1642543604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6820', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6821', 'nonce': 1642543604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6826', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6828', 'nonce': 1642543604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6823', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6824', 'nonce': 1642543604}]}, 'public_addr': '172.21.15.167:6818/1642543604', 'cluster_addr': '172.21.15.167:6821/1642543604', 'heartbeat_back_addr': '172.21.15.167:6828/1642543604', 'heartbeat_front_addr': '172.21.15.167:6824/1642543604', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6808', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 172726194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6810', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 172726194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6814', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 172726194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6812', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 172726194}]}, 'public_addr': '172.21.15.167:6809/172726194', 'cluster_addr': '172.21.15.167:6811/172726194', 'heartbeat_back_addr': '172.21.15.167:6815/172726194', 'heartbeat_front_addr': '172.21.15.167:6813/172726194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6dad12bf-94de-482a-a402-49e67d5e5dc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6800', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2195746987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6802', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2195746987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6806', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2195746987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6804', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2195746987}]}, 'public_addr': '172.21.15.167:6801/2195746987', 'cluster_addr': '172.21.15.167:6803/2195746987', 'heartbeat_back_addr': '172.21.15.167:6807/2195746987', 'heartbeat_front_addr': '172.21.15.167:6805/2195746987', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e6734cfe-3c24-4ddd-8ef0-5ed450efe772', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6817', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6819', 'nonce': 954791674}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6822', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6825', 'nonce': 954791674}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6830', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6831', 'nonce': 954791674}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6827', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6829', 'nonce': 954791674}]}, 'public_addr': '172.21.15.167:6819/954791674', 'cluster_addr': '172.21.15.167:6825/954791674', 'heartbeat_back_addr': '172.21.15.167:6831/954791674', 'heartbeat_front_addr': '172.21.15.167:6829/954791674', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.509233+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.465696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.405177+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.269202+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.359180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.351834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.490127+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.484842+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 4, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T00:34:04.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:34:04.618 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2327 3.19 3.fee2699 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.568084+0000", 2024-04-24T00:34:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0030593949999999999, 2024-04-24T00:34:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030845719999999998, 2024-04-24T00:34:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2327 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568084+0000", 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568084+0000", 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568084+0000", 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568087+0000", 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2189999999999998e-06 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568094+0000", 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3220000000000004e-06 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568101+0000", 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5299999999999999e-06 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2328 3.19 3.fee2699 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.568109+0000", 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.003034751, 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031331660000000002, 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2328 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568109+0000", 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568109+0000", 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568109+0000", 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1800000000000001e-07 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568111+0000", 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.255e-06 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568118+0000", 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.798e-06 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568123+0000", 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2159999999999997e-06 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2329 3.19 3.fee2699 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.568130+0000", 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0030129390000000001, 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031725400000000002, 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2329 2024-04-24T00:34:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568130+0000", 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568130+0000", 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568131+0000", 2024-04-24T00:34:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568133+0000", 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0059999999999999e-06 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568140+0000", 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8410000000000002e-06 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568145+0000", 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1340000000000003e-06 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2330 3.19 3.fee2699 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.568160+0000", 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0029834050000000002, 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032046739999999998, 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2330 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568160+0000", 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568160+0000", 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568160+0000", 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7599999999999997e-07 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568163+0000", 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0350000000000002e-06 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568171+0000", 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8450000000000003e-06 2024-04-24T00:34:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.568177+0000", 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4909999999999996e-06 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 4 2024-04-24T00:34:04.631 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:04.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:04.632 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:04.612+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:04.717 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:04.712+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:04.777 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:04.778 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:04.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:04.837 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:04.832+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:04.919 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:04.916+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:04.982 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2025 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1638400~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44968034200000001, 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077913280000000001, 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2025 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530291+0000", 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3200000000000005e-07 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530294+0000", 2024-04-24T00:34:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530303+0000", 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8820000000000007e-06 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530310+0000", 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0199999999999997e-06 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.537775+0000", 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074650569999999998 2024-04-24T00:34:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.537853+0000", 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7961999999999997e-05 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538082+0000", 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0002288 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2026 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1769472~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44965118199999998, 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081592609999999993, 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2026 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530320+0000", 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6700000000000003e-07 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530323+0000", 2024-04-24T00:34:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7420000000000002e-06 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530332+0000", 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6059999999999992e-06 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530340+0000", 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000001e-06 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538180+0000", 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078402369999999999 2024-04-24T00:34:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538308+0000", 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012834000000000001 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538479+0000", 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170507 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2027 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44962121700000002, 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.009211136, 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2027 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530350+0000", 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7199999999999999e-07 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530353+0000", 2024-04-24T00:34:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3390000000000001e-06 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530362+0000", 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0489999999999993e-06 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530370+0000", 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5739999999999998e-06 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539401+0000", 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090311609999999994 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539466+0000", 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4880000000000002e-05 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539561+0000", 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4561000000000005e-05 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2028 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44959069400000001, 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095022319999999993, 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2028 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7899999999999998e-07 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530384+0000", 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0280000000000001e-06 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530392+0000", 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7830000000000004e-06 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530400+0000", 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2080000000000003e-06 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539616+0000", 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092167310000000006 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539722+0000", 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010527900000000001 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539882+0000", 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000160624 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2045 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44956063099999999, 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097538699999999996, 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2045 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530411+0000", 2024-04-24T00:34:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530414+0000", 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2430000000000001e-06 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530423+0000", 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3030000000000005e-06 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530430+0000", 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.917e-06 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539923+0000", 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094929349999999992 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540027+0000", 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010355 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540164+0000", 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013730799999999999 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2046 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44953042500000001, 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010030647, 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2046 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530441+0000", 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3200000000000005e-07 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530444+0000", 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8109999999999999e-06 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530453+0000", 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3409999999999992e-06 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530460+0000", 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3719999999999999e-06 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540209+0000", 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097491539999999995 2024-04-24T00:34:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540326+0000", 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001166 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540471+0000", 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014483700000000001 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2047 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44949975599999997, 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010295987, 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2047 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530472+0000", 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2799999999999995e-07 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530475+0000", 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0939999999999999e-06 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530485+0000", 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7969999999999995e-06 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530491+0000", 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8709999999999999e-06 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540514+0000", 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010022499000000001 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540626+0000", 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111981 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540767+0000", 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014091700000000001 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2048 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44946894500000001, 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010591068, 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2048 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2699999999999999e-07 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530520+0000", 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8165999999999999e-05 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530533+0000", 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2065999999999999e-05 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540812+0000", 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010266239 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540925+0000", 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112676 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541093+0000", 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016786500000000001 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2049 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.449412702, 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010815047, 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2049 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530562+0000", 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4369999999999999e-06 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530572+0000", 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530579+0000", 2024-04-24T00:34:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541138+0000", 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010558209000000001 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541239+0000", 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101483 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541373+0000", 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133769 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2050 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44938135299999998, 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011052364, 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2050 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530590+0000", 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6599999999999996e-07 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530593+0000", 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.937e-06 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530602+0000", 2024-04-24T00:34:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4360000000000008e-06 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530609+0000", 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0090000000000001e-06 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541411+0000", 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01080191 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541511+0000", 2024-04-24T00:34:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9839000000000004e-05 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541642+0000", 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130667 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2051 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44935199300000001, 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011318002000000001, 2024-04-24T00:34:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2051 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2900000000000004e-07 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530622+0000", 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9490000000000001e-06 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530632+0000", 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999994e-06 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.131e-06 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541683+0000", 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011044629 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120383 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541937+0000", 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132941 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2052 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.449322793, 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011583872, 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2052 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.44e-07 2024-04-24T00:34:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530658+0000", 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9850000000000001e-06 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530668+0000", 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530675+0000", 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6240000000000002e-06 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541978+0000", 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011302921 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542100+0000", 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121593 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542232+0000", 2024-04-24T00:34:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013176600000000001 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2077 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44928539299999998, 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011842221, 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2077 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6000000000000003e-07 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.095e-06 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530699+0000", 2024-04-24T00:34:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530706+0000", 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2429999999999999e-06 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542275+0000", 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011568746 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542386+0000", 2024-04-24T00:34:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011176200000000001 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542527+0000", 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014114499999999999 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2078 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44925506700000001, 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012081010999999999, 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2078 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2200000000000004e-07 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530719+0000", 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9349999999999999e-06 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530728+0000", 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0860000000000008e-06 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530735+0000", 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2169999999999997e-06 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542568+0000", 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011832540000000001 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542667+0000", 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9000999999999996e-05 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542797+0000", 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012971 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2079 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44922575399999998, 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012334128999999999, 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2079 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530748+0000", 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9040000000000001e-06 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2159999999999995e-06 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530764+0000", 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9890000000000003e-06 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542839+0000", 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012074354000000001 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542945+0000", 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106443 2024-04-24T00:34:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543079+0000", 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013369599999999999 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2080 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44919695399999998, 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012588825999999999, 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2080 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530777+0000", 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.022e-06 2024-04-24T00:34:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9879999999999993e-06 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530793+0000", 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1409999999999999e-06 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543122+0000", 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012328274 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543230+0000", 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108046 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543362+0000", 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013286799999999999 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2081 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.449168069, 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012843515, 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2081 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1500000000000005e-07 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.819e-06 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530815+0000", 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3540000000000006e-06 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530835+0000", 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9983000000000002e-05 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543405+0000", 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012569739 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543513+0000", 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010821500000000001 2024-04-24T00:34:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543646+0000", 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289000000000001 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2082 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44912485400000002, 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013081859, 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2082 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530849+0000", 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0750000000000002e-06 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530859+0000", 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7079999999999992e-06 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530866+0000", 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4309999999999997e-06 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543689+0000", 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012822750000000001 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543796+0000", 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106526 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543928+0000", 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131846 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2083 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44909458099999999, 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013453964000000001, 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2083 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530877+0000", 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530887+0000", 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186e-05 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530896+0000", 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5480000000000007e-06 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530904+0000", 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4599999999999997e-06 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543970+0000", 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013066067000000001 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544178+0000", 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020785000000000001 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544330+0000", 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015230699999999999 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2084 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.44905675699999997, 2024-04-24T00:34:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907047000000001, 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2084 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3099999999999998e-07 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530922+0000", 2024-04-24T00:34:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.498e-06 2024-04-24T00:34:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530932+0000", 2024-04-24T00:34:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-04-24T00:34:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530940+0000", 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4769999999999997e-06 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544421+0000", 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013481739 2024-04-24T00:34:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544673+0000", 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025143600000000001 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544821+0000", 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014819 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.070 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:05.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:34:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:34:05.075 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-24T00:34:04.529642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":143593988,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":3297,"num_read_kb":4199,"num_write":971,"num_write_kb":7464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":813,"ondisk_log_size":813,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":205096,"kb_used_data":3384,"kb_used_omap":10,"kb_used_meta":201525,"kb_avail":660397784,"statfs":{"total":676457349120,"available":676247330816,"internally_reserved":0,"allocated":3465216,"data_stored":3494957,"data_compressed":502262,"data_compressed_allocated":647168,"data_compressed_original":2367488,"omap_allocated":11157,"internal_metadata":206361707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":137192650,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1880,"num_read_kb":2876,"num_write":503,"num_write_kb":3329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.072684"},"pg_stats":[{"pgid":"3.1f","version":"31'23","reported_seq":83,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.536695+0000","last_change":"2024-04-24T00:34:00.471661+0000","last_active":"2024-04-24T00:34:03.536695+0000","last_peered":"2024-04-24T00:34:03.536695+0000","last_clean":"2024-04-24T00:34:03.536695+0000","last_became_active":"2024-04-24T00:33:53.021533+0000","last_became_peered":"2024-04-24T00:33:53.021533+0000","last_unstale":"2024-04-24T00:34:03.536695+0000","last_undegraded":"2024-04-24T00:34:03.536695+0000","last_fullsized":"2024-04-24T00:34:03.536695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:25:38.555409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001281237,"stat_sum":{"num_bytes":16507277,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":286,"num_write":22,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.480337+0000","last_change":"2024-04-24T00:34:00.459177+0000","last_active":"2024-04-24T00:34:03.480337+0000","last_peered":"2024-04-24T00:34:03.480337+0000","last_clean":"2024-04-24T00:34:03.480337+0000","last_became_active":"2024-04-24T00:33:53.018728+0000","last_became_peered":"2024-04-24T00:33:53.018728+0000","last_unstale":"2024-04-24T00:34:03.480337+0000","last_undegraded":"2024-04-24T00:34:03.480337+0000","last_fullsized":"2024-04-24T00:34:03.480337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:43:26.175062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012358130000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.478485+0000","last_change":"2024-04-24T00:34:00.460749+0000","last_active":"2024-04-24T00:34:03.478485+0000","last_peered":"2024-04-24T00:34:03.478485+0000","last_clean":"2024-04-24T00:34:03.478485+0000","last_became_active":"2024-04-24T00:33:53.032813+0000","last_became_peered":"2024-04-24T00:33:53.032813+0000","last_unstale":"2024-04-24T00:34:03.478485+0000","last_undegraded":"2024-04-24T00:34:03.478485+0000","last_fullsized":"2024-04-24T00:34:03.478485+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:31:39.589509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024009209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.1c","version":"31'5","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.478467+0000","last_change":"2024-04-24T00:34:00.459696+0000","last_active":"2024-04-24T00:34:03.478467+0000","last_peered":"2024-04-24T00:34:03.478467+0000","last_clean":"2024-04-24T00:34:03.478467+0000","last_became_active":"2024-04-24T00:33:53.029450+0000","last_became_peered":"2024-04-24T00:33:53.029450+0000","last_unstale":"2024-04-24T00:34:03.478467+0000","last_undegraded":"2024-04-24T00:34:03.478467+0000","last_fullsized":"2024-04-24T00:34:03.478467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:42:26.883745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013403759999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.1b","version":"25'10","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.487729+0000","last_change":"2024-04-24T00:34:00.473454+0000","last_active":"2024-04-24T00:34:03.487729+0000","last_peered":"2024-04-24T00:34:03.487729+0000","last_clean":"2024-04-24T00:34:03.487729+0000","last_became_active":"2024-04-24T00:33:53.035615+0000","last_became_peered":"2024-04-24T00:33:53.035615+0000","last_unstale":"2024-04-24T00:34:03.487729+0000","last_undegraded":"2024-04-24T00:34:03.487729+0000","last_fullsized":"2024-04-24T00:34:03.487729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:31:55.886063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022453199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.1a","version":"31'9","reported_seq":154,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.512660+0000","last_change":"2024-04-24T00:34:00.459493+0000","last_active":"2024-04-24T00:34:03.512660+0000","last_peered":"2024-04-24T00:34:03.512660+0000","last_clean":"2024-04-24T00:34:03.512660+0000","last_became_active":"2024-04-24T00:33:53.027055+0000","last_became_peered":"2024-04-24T00:33:53.027055+0000","last_unstale":"2024-04-24T00:34:03.512660+0000","last_undegraded":"2024-04-24T00:34:03.512660+0000","last_fullsized":"2024-04-24T00:34:03.512660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:06:37.054353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014156189999999999,"stat_sum":{"num_bytes":104,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":104,"num_write":9,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.19","version":"31'22","reported_seq":74,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.535781+0000","last_change":"2024-04-24T00:34:00.462820+0000","last_active":"2024-04-24T00:34:03.535781+0000","last_peered":"2024-04-24T00:34:03.535781+0000","last_clean":"2024-04-24T00:34:03.535781+0000","last_became_active":"2024-04-24T00:33:53.017357+0000","last_became_peered":"2024-04-24T00:33:53.017357+0000","last_unstale":"2024-04-24T00:34:03.535781+0000","last_undegraded":"2024-04-24T00:34:03.535781+0000","last_fullsized":"2024-04-24T00:34:03.535781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:44:38.841601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020996919999999998,"stat_sum":{"num_bytes":9437248,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":75,"num_write":20,"num_write_kb":294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.18","version":"29'4","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.479305+0000","last_change":"2024-04-24T00:34:00.458594+0000","last_active":"2024-04-24T00:34:03.479305+0000","last_peered":"2024-04-24T00:34:03.479305+0000","last_clean":"2024-04-24T00:34:03.479305+0000","last_became_active":"2024-04-24T00:33:53.019129+0000","last_became_peered":"2024-04-24T00:33:53.019129+0000","last_unstale":"2024-04-24T00:34:03.479305+0000","last_undegraded":"2024-04-24T00:34:03.479305+0000","last_fullsized":"2024-04-24T00:34:03.479305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:46:20.416308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070233600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.17","version":"30'19","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.488122+0000","last_change":"2024-04-24T00:34:00.472064+0000","last_active":"2024-04-24T00:34:03.488122+0000","last_peered":"2024-04-24T00:34:03.488122+0000","last_clean":"2024-04-24T00:34:03.488122+0000","last_became_active":"2024-04-24T00:33:53.018638+0000","last_became_peered":"2024-04-24T00:33:53.018638+0000","last_unstale":"2024-04-24T00:34:03.488122+0000","last_undegraded":"2024-04-24T00:34:03.488122+0000","last_fullsized":"2024-04-24T00:34:03.488122+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:26:16.799542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001357631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":234,"num_write":19,"num_write_kb":191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.16","version":"28'51","reported_seq":343,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.491134+0000","last_change":"2024-04-24T00:34:00.477665+0000","last_active":"2024-04-24T00:34:03.491134+0000","last_peered":"2024-04-24T00:34:03.491134+0000","last_clean":"2024-04-24T00:34:03.491134+0000","last_became_active":"2024-04-24T00:33:53.025504+0000","last_became_peered":"2024-04-24T00:33:53.025504+0000","last_unstale":"2024-04-24T00:34:03.491134+0000","last_undegraded":"2024-04-24T00:34:03.491134+0000","last_fullsized":"2024-04-24T00:34:03.491134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:58:37.581706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021809160000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":708,"num_read_kb":622,"num_write":47,"num_write_kb":169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.15","version":"25'8","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.490851+0000","last_change":"2024-04-24T00:34:00.475745+0000","last_active":"2024-04-24T00:34:03.490851+0000","last_peered":"2024-04-24T00:34:03.490851+0000","last_clean":"2024-04-24T00:34:03.490851+0000","last_became_active":"2024-04-24T00:33:53.038882+0000","last_became_peered":"2024-04-24T00:33:53.038882+0000","last_unstale":"2024-04-24T00:34:03.490851+0000","last_undegraded":"2024-04-24T00:34:03.490851+0000","last_fullsized":"2024-04-24T00:34:03.490851+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:29.765800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013349530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.14","version":"0'0","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.478462+0000","last_change":"2024-04-24T00:34:00.460251+0000","last_active":"2024-04-24T00:34:03.478462+0000","last_peered":"2024-04-24T00:34:03.478462+0000","last_clean":"2024-04-24T00:34:03.478462+0000","last_became_active":"2024-04-24T00:33:53.032030+0000","last_became_peered":"2024-04-24T00:33:53.032030+0000","last_unstale":"2024-04-24T00:34:03.478462+0000","last_undegraded":"2024-04-24T00:34:03.478462+0000","last_fullsized":"2024-04-24T00:34:03.478462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:50.853308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015520410000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.481813+0000","last_change":"2024-04-24T00:33:44.961547+0000","last_active":"2024-04-24T00:34:03.481813+0000","last_peered":"2024-04-24T00:34:03.481813+0000","last_clean":"2024-04-24T00:34:03.481813+0000","last_became_active":"2024-04-24T00:33:44.960683+0000","last_became_peered":"2024-04-24T00:33:44.960683+0000","last_unstale":"2024-04-24T00:34:03.481813+0000","last_undegraded":"2024-04-24T00:34:03.481813+0000","last_fullsized":"2024-04-24T00:34:03.481813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"30'16","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.487824+0000","last_change":"2024-04-24T00:34:00.471935+0000","last_active":"2024-04-24T00:34:03.487824+0000","last_peered":"2024-04-24T00:34:03.487824+0000","last_clean":"2024-04-24T00:34:03.487824+0000","last_became_active":"2024-04-24T00:33:53.017081+0000","last_became_peered":"2024-04-24T00:33:53.017081+0000","last_unstale":"2024-04-24T00:34:03.487824+0000","last_undegraded":"2024-04-24T00:34:03.487824+0000","last_fullsized":"2024-04-24T00:34:03.487824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:46:30.585629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001217138,"stat_sum":{"num_bytes":15728640,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":55,"num_write":14,"num_write_kb":202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.481001+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:34:03.481001+0000","last_peered":"2024-04-24T00:34:03.481001+0000","last_clean":"2024-04-24T00:34:03.481001+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:34:03.481001+0000","last_undegraded":"2024-04-24T00:34:03.481001+0000","last_fullsized":"2024-04-24T00:34:03.481001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.481130+0000","last_change":"2024-04-24T00:34:00.459472+0000","last_active":"2024-04-24T00:34:03.481130+0000","last_peered":"2024-04-24T00:34:03.481130+0000","last_clean":"2024-04-24T00:34:03.481130+0000","last_became_active":"2024-04-24T00:33:53.038083+0000","last_became_peered":"2024-04-24T00:33:53.038083+0000","last_unstale":"2024-04-24T00:34:03.481130+0000","last_undegraded":"2024-04-24T00:34:03.481130+0000","last_fullsized":"2024-04-24T00:34:03.481130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:28.487597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012000489999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.491019+0000","last_change":"2024-04-24T00:33:45.648584+0000","last_active":"2024-04-24T00:34:03.491019+0000","last_peered":"2024-04-24T00:34:03.491019+0000","last_clean":"2024-04-24T00:34:03.491019+0000","last_became_active":"2024-04-24T00:33:45.647889+0000","last_became_peered":"2024-04-24T00:33:45.647889+0000","last_unstale":"2024-04-24T00:34:03.491019+0000","last_undegraded":"2024-04-24T00:34:03.491019+0000","last_fullsized":"2024-04-24T00:34:03.491019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:52.085162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"30'13","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.480885+0000","last_change":"2024-04-24T00:34:00.463304+0000","last_active":"2024-04-24T00:34:03.480885+0000","last_peered":"2024-04-24T00:34:03.480885+0000","last_clean":"2024-04-24T00:34:03.480885+0000","last_became_active":"2024-04-24T00:33:53.015167+0000","last_became_peered":"2024-04-24T00:33:53.015167+0000","last_unstale":"2024-04-24T00:34:03.480885+0000","last_undegraded":"2024-04-24T00:34:03.480885+0000","last_fullsized":"2024-04-24T00:34:03.480885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:07:28.631297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020546050000000001,"stat_sum":{"num_bytes":9361814,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":46,"num_write":13,"num_write_kb":221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.481201+0000","last_change":"2024-04-24T00:33:45.650424+0000","last_active":"2024-04-24T00:34:03.481201+0000","last_peered":"2024-04-24T00:34:03.481201+0000","last_clean":"2024-04-24T00:34:03.481201+0000","last_became_active":"2024-04-24T00:33:45.649310+0000","last_became_peered":"2024-04-24T00:33:45.649310+0000","last_unstale":"2024-04-24T00:34:03.481201+0000","last_undegraded":"2024-04-24T00:34:03.481201+0000","last_fullsized":"2024-04-24T00:34:03.481201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:40:40.748700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"30'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.480273+0000","last_change":"2024-04-24T00:34:00.459871+0000","last_active":"2024-04-24T00:34:03.480273+0000","last_peered":"2024-04-24T00:34:03.480273+0000","last_clean":"2024-04-24T00:34:03.480273+0000","last_became_active":"2024-04-24T00:33:53.016877+0000","last_became_peered":"2024-04-24T00:33:53.016877+0000","last_unstale":"2024-04-24T00:34:03.480273+0000","last_undegraded":"2024-04-24T00:34:03.480273+0000","last_fullsized":"2024-04-24T00:34:03.480273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:11:27.590235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015486339999999999,"stat_sum":{"num_bytes":25068764,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":100,"num_write":34,"num_write_kb":476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.3","version":"24'3","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.478391+0000","last_change":"2024-04-24T00:34:00.460142+0000","last_active":"2024-04-24T00:34:03.478391+0000","last_peered":"2024-04-24T00:34:03.478391+0000","last_clean":"2024-04-24T00:34:03.478391+0000","last_became_active":"2024-04-24T00:33:53.031528+0000","last_became_peered":"2024-04-24T00:33:53.031528+0000","last_unstale":"2024-04-24T00:34:03.478391+0000","last_undegraded":"2024-04-24T00:34:03.478391+0000","last_fullsized":"2024-04-24T00:34:03.478391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:27:52.062038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00169107,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.480767+0000","last_change":"2024-04-24T00:33:53.019966+0000","last_active":"2024-04-24T00:34:03.480767+0000","last_peered":"2024-04-24T00:34:03.480767+0000","last_clean":"2024-04-24T00:34:03.480767+0000","last_became_active":"2024-04-24T00:33:53.019543+0000","last_became_peered":"2024-04-24T00:33:53.019543+0000","last_unstale":"2024-04-24T00:34:03.480767+0000","last_undegraded":"2024-04-24T00:34:03.480767+0000","last_fullsized":"2024-04-24T00:34:03.480767+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:38:43.125607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'1","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.525929+0000","last_change":"2024-04-24T00:34:00.462921+0000","last_active":"2024-04-24T00:34:03.525929+0000","last_peered":"2024-04-24T00:34:03.525929+0000","last_clean":"2024-04-24T00:34:03.525929+0000","last_became_active":"2024-04-24T00:33:53.012086+0000","last_became_peered":"2024-04-24T00:33:53.012086+0000","last_unstale":"2024-04-24T00:34:03.525929+0000","last_undegraded":"2024-04-24T00:34:03.525929+0000","last_fullsized":"2024-04-24T00:34:03.525929+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:54:50.434716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020892039999999999,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.518893+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:34:03.518893+0000","last_peered":"2024-04-24T00:34:03.518893+0000","last_clean":"2024-04-24T00:34:03.518893+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:34:03.518893+0000","last_undegraded":"2024-04-24T00:34:03.518893+0000","last_fullsized":"2024-04-24T00:34:03.518893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"27'12","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.487176+0000","last_change":"2024-04-24T00:34:00.470790+0000","last_active":"2024-04-24T00:34:03.487176+0000","last_peered":"2024-04-24T00:34:03.487176+0000","last_clean":"2024-04-24T00:34:03.487176+0000","last_became_active":"2024-04-24T00:33:53.032926+0000","last_became_peered":"2024-04-24T00:33:53.032926+0000","last_unstale":"2024-04-24T00:34:03.487176+0000","last_undegraded":"2024-04-24T00:34:03.487176+0000","last_fullsized":"2024-04-24T00:34:03.487176+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:12:17.102571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000547464,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":52,"num_write":18,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.490553+0000","last_change":"2024-04-24T00:33:44.961018+0000","last_active":"2024-04-24T00:34:03.490553+0000","last_peered":"2024-04-24T00:34:03.490553+0000","last_clean":"2024-04-24T00:34:03.490553+0000","last_became_active":"2024-04-24T00:33:44.959946+0000","last_became_peered":"2024-04-24T00:33:44.959946+0000","last_unstale":"2024-04-24T00:34:03.490553+0000","last_undegraded":"2024-04-24T00:34:03.490553+0000","last_fullsized":"2024-04-24T00:34:03.490553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:12.966651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"30'22","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.479356+0000","last_change":"2024-04-24T00:34:00.459106+0000","last_active":"2024-04-24T00:34:03.479356+0000","last_peered":"2024-04-24T00:34:03.479356+0000","last_clean":"2024-04-24T00:34:03.479356+0000","last_became_active":"2024-04-24T00:33:53.019204+0000","last_became_peered":"2024-04-24T00:33:53.019204+0000","last_unstale":"2024-04-24T00:34:03.479356+0000","last_undegraded":"2024-04-24T00:34:03.479356+0000","last_fullsized":"2024-04-24T00:34:03.479356+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:55:52.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012015979999999999,"stat_sum":{"num_bytes":15728656,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":143,"num_write":21,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.518732+0000","last_change":"2024-04-24T00:33:53.018981+0000","last_active":"2024-04-24T00:34:03.518732+0000","last_peered":"2024-04-24T00:34:03.518732+0000","last_clean":"2024-04-24T00:34:03.518732+0000","last_became_active":"2024-04-24T00:33:53.018648+0000","last_became_peered":"2024-04-24T00:33:53.018648+0000","last_unstale":"2024-04-24T00:34:03.518732+0000","last_undegraded":"2024-04-24T00:34:03.518732+0000","last_fullsized":"2024-04-24T00:34:03.518732+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:17.520316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.490570+0000","last_change":"2024-04-24T00:33:41.026859+0000","last_active":"2024-04-24T00:34:03.490570+0000","last_peered":"2024-04-24T00:34:03.490570+0000","last_clean":"2024-04-24T00:34:03.490570+0000","last_became_active":"2024-04-24T00:33:41.024668+0000","last_became_peered":"2024-04-24T00:33:41.024668+0000","last_unstale":"2024-04-24T00:34:03.490570+0000","last_undegraded":"2024-04-24T00:34:03.490570+0000","last_fullsized":"2024-04-24T00:34:03.490570+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:25.500752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"24'3","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.479514+0000","last_change":"2024-04-24T00:34:00.460023+0000","last_active":"2024-04-24T00:34:03.479514+0000","last_peered":"2024-04-24T00:34:03.479514+0000","last_clean":"2024-04-24T00:34:03.479514+0000","last_became_active":"2024-04-24T00:33:53.022355+0000","last_became_peered":"2024-04-24T00:33:53.022355+0000","last_unstale":"2024-04-24T00:34:03.479514+0000","last_undegraded":"2024-04-24T00:34:03.479514+0000","last_fullsized":"2024-04-24T00:34:03.479514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:44.265202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001561462,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.9","version":"30'26","reported_seq":78,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.478268+0000","last_change":"2024-04-24T00:34:00.459035+0000","last_active":"2024-04-24T00:34:03.478268+0000","last_peered":"2024-04-24T00:34:03.478268+0000","last_clean":"2024-04-24T00:34:03.478268+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:34:03.478268+0000","last_undegraded":"2024-04-24T00:34:03.478268+0000","last_fullsized":"2024-04-24T00:34:03.478268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:19.304048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010258349999999999,"stat_sum":{"num_bytes":16622560,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":331,"num_write":22,"num_write_kb":358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.481881+0000","last_change":"2024-04-24T00:34:00.465876+0000","last_active":"2024-04-24T00:34:03.481881+0000","last_peered":"2024-04-24T00:34:03.481881+0000","last_clean":"2024-04-24T00:34:03.481881+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:34:03.481881+0000","last_undegraded":"2024-04-24T00:34:03.481881+0000","last_fullsized":"2024-04-24T00:34:03.481881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:46.191246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079943099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.b","version":"25'10","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.479848+0000","last_change":"2024-04-24T00:34:00.459602+0000","last_active":"2024-04-24T00:34:03.479848+0000","last_peered":"2024-04-24T00:34:03.479848+0000","last_clean":"2024-04-24T00:34:03.479848+0000","last_became_active":"2024-04-24T00:33:53.019011+0000","last_became_peered":"2024-04-24T00:33:53.019011+0000","last_unstale":"2024-04-24T00:34:03.479848+0000","last_undegraded":"2024-04-24T00:34:03.479848+0000","last_fullsized":"2024-04-24T00:34:03.479848+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:32:16.689366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013454280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.c","version":"30'126","reported_seq":175,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.479872+0000","last_change":"2024-04-24T00:34:00.458606+0000","last_active":"2024-04-24T00:34:03.479872+0000","last_peered":"2024-04-24T00:34:03.479872+0000","last_clean":"2024-04-24T00:34:03.479872+0000","last_became_active":"2024-04-24T00:33:53.024773+0000","last_became_peered":"2024-04-24T00:33:53.024773+0000","last_unstale":"2024-04-24T00:34:03.479872+0000","last_undegraded":"2024-04-24T00:34:03.479872+0000","last_fullsized":"2024-04-24T00:34:03.479872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:40.868413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069536800000000003,"stat_sum":{"num_bytes":18515329,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":500,"num_read_kb":430,"num_write":188,"num_write_kb":522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.490957+0000","last_change":"2024-04-24T00:34:00.476395+0000","last_active":"2024-04-24T00:34:03.490957+0000","last_peered":"2024-04-24T00:34:03.490957+0000","last_clean":"2024-04-24T00:34:03.490957+0000","last_became_active":"2024-04-24T00:33:53.028960+0000","last_became_peered":"2024-04-24T00:33:53.028960+0000","last_unstale":"2024-04-24T00:34:03.490957+0000","last_undegraded":"2024-04-24T00:34:03.490957+0000","last_fullsized":"2024-04-24T00:34:03.490957+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:41:29.034369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015200579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.e","version":"25'3","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.491253+0000","last_change":"2024-04-24T00:34:00.477087+0000","last_active":"2024-04-24T00:34:03.491253+0000","last_peered":"2024-04-24T00:34:03.491253+0000","last_clean":"2024-04-24T00:34:03.491253+0000","last_became_active":"2024-04-24T00:33:53.031663+0000","last_became_peered":"2024-04-24T00:33:53.031663+0000","last_unstale":"2024-04-24T00:34:03.491253+0000","last_undegraded":"2024-04-24T00:34:03.491253+0000","last_fullsized":"2024-04-24T00:34:03.491253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:20.153492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018251420000000001,"stat_sum":{"num_bytes":3024147,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":35,"num_write":3,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.f","version":"24'4","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.491428+0000","last_change":"2024-04-24T00:34:00.476427+0000","last_active":"2024-04-24T00:34:03.491428+0000","last_peered":"2024-04-24T00:34:03.491428+0000","last_clean":"2024-04-24T00:34:03.491428+0000","last_became_active":"2024-04-24T00:33:53.029304+0000","last_became_peered":"2024-04-24T00:33:53.029304+0000","last_unstale":"2024-04-24T00:34:03.491428+0000","last_undegraded":"2024-04-24T00:34:03.491428+0000","last_fullsized":"2024-04-24T00:34:03.491428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:23:18.455076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020033080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.10","version":"31'75","reported_seq":496,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.507920+0000","last_change":"2024-04-24T00:34:00.467260+0000","last_active":"2024-04-24T00:34:03.507920+0000","last_peered":"2024-04-24T00:34:03.507920+0000","last_clean":"2024-04-24T00:34:03.507920+0000","last_became_active":"2024-04-24T00:33:53.019812+0000","last_became_peered":"2024-04-24T00:33:53.019812+0000","last_unstale":"2024-04-24T00:34:03.507920+0000","last_undegraded":"2024-04-24T00:34:03.507920+0000","last_fullsized":"2024-04-24T00:34:03.507920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:11.487804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025891640000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1095,"num_read_kb":962,"num_write":76,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.11","version":"30'59","reported_seq":105,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.490677+0000","last_change":"2024-04-24T00:34:00.475112+0000","last_active":"2024-04-24T00:34:03.490677+0000","last_peered":"2024-04-24T00:34:03.490677+0000","last_clean":"2024-04-24T00:34:03.490677+0000","last_became_active":"2024-04-24T00:33:53.026709+0000","last_became_peered":"2024-04-24T00:33:53.026709+0000","last_unstale":"2024-04-24T00:34:03.490677+0000","last_undegraded":"2024-04-24T00:34:03.490677+0000","last_fullsized":"2024-04-24T00:34:03.490677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:35.270442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008739,"stat_sum":{"num_bytes":13009025,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":150,"num_write":54,"num_write_kb":735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.12","version":"31'134","reported_seq":178,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.492080+0000","last_change":"2024-04-24T00:34:00.473314+0000","last_active":"2024-04-24T00:34:03.492080+0000","last_peered":"2024-04-24T00:34:03.492080+0000","last_clean":"2024-04-24T00:34:03.492080+0000","last_became_active":"2024-04-24T00:33:53.026957+0000","last_became_peered":"2024-04-24T00:33:53.026957+0000","last_unstale":"2024-04-24T00:34:03.492080+0000","last_undegraded":"2024-04-24T00:34:03.492080+0000","last_fullsized":"2024-04-24T00:34:03.492080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:03.680111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025617790000000001,"stat_sum":{"num_bytes":40,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":596,"num_read_kb":467,"num_write":202,"num_write_kb":258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T00:34:03.490764+0000","last_change":"2024-04-24T00:34:00.475194+0000","last_active":"2024-04-24T00:34:03.490764+0000","last_peered":"2024-04-24T00:34:03.490764+0000","last_clean":"2024-04-24T00:34:03.490764+0000","last_became_active":"2024-04-24T00:33:53.033430+0000","last_became_peered":"2024-04-24T00:33:53.033430+0000","last_unstale":"2024-04-24T00:34:03.490764+0000","last_undegraded":"2024-04-24T00:34:03.490764+0000","last_fullsized":"2024-04-24T00:34:03.490764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:30:35.251564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"},{"start":"4","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":143003620,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":3181,"num_read_kb":4099,"num_write":812,"num_write_kb":5048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1032192,"data_stored":1396954,"data_compressed":490014,"data_compressed_allocated":573440,"data_compressed_original":1220608,"omap_allocated":0,"internal_metadata":0},"log_size":696,"ondisk_log_size":696,"up":64,"acting":64,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12248,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31604,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340236,"statfs":{"total":96636764160,"available":96604401664,"internally_reserved":0,"allocated":438272,"data_stored":835138,"data_compressed":6361,"data_compressed_allocated":49152,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":304182,"data_compressed":65249,"data_compressed_allocated":73728,"data_compressed_original":151552,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":55834574856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":442368,"data_stored":307169,"data_compressed":35521,"data_compressed_allocated":49152,"data_compressed_original":151552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31788,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":524288,"data_stored":406103,"data_compressed":92658,"data_compressed_allocated":110592,"data_compressed_original":245760,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":589824,"data_stored":470889,"data_compressed":138793,"data_compressed_allocated":159744,"data_compressed_original":299008,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":491520,"data_stored":295599,"data_compressed":95033,"data_compressed_allocated":98304,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":516096,"data_stored":875877,"data_compressed":68647,"data_compressed_allocated":106496,"data_compressed_original":704512,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":150932,"data_compressed":62523,"data_compressed_allocated":69632,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":161022,"data_compressed":95033,"data_compressed_allocated":98304,"data_compressed_original":143360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":342491,"data_compressed":138793,"data_compressed_allocated":159744,"data_compressed_original":299008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":277753,"data_compressed":92658,"data_compressed_allocated":110592,"data_compressed_original":245760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":178795,"data_compressed":35521,"data_compressed_allocated":49152,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":169605,"data_compressed":65249,"data_compressed_allocated":73728,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":116356,"data_compressed":237,"data_compressed_allocated":12288,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:34:05.077 INFO:tasks.thrashosds.thrasher:Setting 3.1b to [5, 6] 2024-04-24T00:34:05.077 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1b', '5', '6'] 2024-04-24T00:34:05.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.1b 5 6 2024-04-24T00:34:05.077 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:05.068+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:05.173 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:05.168+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:05.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:05.235 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:05.235 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:05.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:05.274 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:05.268+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:05.374 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:05.368+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:05.406 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:05.406 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:05.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:05.475 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:05.472+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:set 3.1b pg_upmap mapping to [5,6] 2024-04-24T00:34:05.574 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:05.575 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:05.568+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:11 3.1f e24/21 3:fa654f18:::rbd_id.image_client.0:head v 24'1, mlcod=0'0)", 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.546449000999999, 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001106349, 2024-04-24T00:34:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.06e-07 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037959+0000", 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4007e-05 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037974+0000", 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4586999999999999e-05 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037985+0000", 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.137e-05 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.038111+0000", 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012570699999999999 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.038143+0000", 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2218999999999998e-05 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039003+0000", 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085975500000000002 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039051+0000", 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8097999999999999e-05 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.591 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1717 3.0 e31/21 3:04479396:::rbd_id.image_client.0-clone2:head v 31'1, mlcod=0'0)", 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:03.517225+0000", 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0671686600000001, 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001023382, 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517225+0000", 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517225+0000", 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517225+0000", 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7000000000000001e-07 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517231+0000", 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5790000000000003e-06 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517246+0000", 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5204e-05 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517258+0000", 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1552000000000001e-05 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517322+0000", 2024-04-24T00:34:05.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4103000000000006e-05 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517345+0000", 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3212000000000001e-05 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.518195+0000", 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085033099999999996 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.518249+0000", 2024-04-24T00:34:05.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3130999999999997e-05 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1718 3.1f e31/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 31'20, mlcod=30'18)", 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:03.519287+0000", 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0651068669999999, 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061245699999999995, 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519287+0000", 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519287+0000", 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519287+0000", 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9900000000000002e-07 2024-04-24T00:34:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519291+0000", 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7400000000000002e-06 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519302+0000", 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0998e-05 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519308+0000", 2024-04-24T00:34:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519348+0000", 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9804e-05 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519361+0000", 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.29e-05 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519856+0000", 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049467499999999998 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.519899+0000", 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3707e-05 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1719 3.1f e31/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 31'21, mlcod=30'19)", 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:03.520684+0000", 2024-04-24T00:34:05.600 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.063709496, 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075079400000000005, 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.520684+0000", 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.520684+0000", 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.520685+0000", 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.520691+0000", 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.520706+0000", 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4226999999999999e-05 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.520726+0000", 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0012000000000001e-05 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.520794+0000", 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8045000000000003e-05 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.520819+0000", 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5103e-05 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.521393+0000", 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057456900000000003 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.521435+0000", 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1823000000000003e-05 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1733 3.1f e31/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 31'22, mlcod=31'20)", 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:03.531461+0000", 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.052932464, 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072118799999999999, 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.531461+0000", 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.531461+0000", 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.531463+0000", 2024-04-24T00:34:05.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.531472+0000", 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5569999999999996e-06 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.531491+0000", 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9483e-05 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.531503+0000", 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1943000000000001e-05 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.531582+0000", 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8603999999999995e-05 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.531608+0000", 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6299999999999999e-05 2024-04-24T00:34:05.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.532144+0000", 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053554700000000004 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.532183+0000", 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8791000000000002e-05 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1735 3.1f e31/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 31'23, mlcod=31'21)", 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:03.533332+0000", 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0510618840000001, 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069927900000000002, 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533332+0000", 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533332+0000", 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533333+0000", 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5499999999999997e-07 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533341+0000", 2024-04-24T00:34:05.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6580000000000002e-06 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533355+0000", 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.401e-05 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533361+0000", 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7499999999999997e-06 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533426+0000", 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4720000000000004e-05 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533449+0000", 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2938999999999998e-05 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.533987+0000", 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053799700000000002 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.534031+0000", 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4350000000000001e-05 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1741 3.1f e32/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 32'24, mlcod=31'22)", 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.501543+0000", 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0828505429999999, 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083306799999999996, 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.614 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501543+0000", 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501543+0000", 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501544+0000", 2024-04-24T00:34:05.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4799999999999998e-07 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501553+0000", 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0219999999999993e-06 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501582+0000", 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501600+0000", 2024-04-24T00:34:05.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8363999999999999e-05 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501681+0000", 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0863000000000002e-05 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501705+0000", 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4060999999999999e-05 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.502329+0000", 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062386799999999997 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.502376+0000", 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7172000000000002e-05 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1745 3.1f e32/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 32'25, mlcod=31'23)", 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.513069+0000", 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0713244019999999, 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074451599999999997, 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513069+0000", 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513069+0000", 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513071+0000", 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.482e-06 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513081+0000", 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.978e-06 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513100+0000", 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9463e-05 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513110+0000", 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513181+0000", 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0839000000000004e-05 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513204+0000", 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3294000000000001e-05 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513768+0000", 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056432400000000005 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.513814+0000", 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5697e-05 2024-04-24T00:34:05.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2348 3.1f e33/21 3:fac32ee7:::rbd_data.115e5ff16b0c.0000000000000009:head v 33'26, mlcod=32'24)", 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.011606507, 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011498890000000001, 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572788+0000", 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572816+0000", 2024-04-24T00:34:05.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8061000000000001e-05 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572827+0000", 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1423999999999999e-05 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572835+0000", 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.858e-06 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572898+0000", 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3139000000000007e-05 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572929+0000", 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0355999999999999e-05 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.573887+0000", 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095867599999999999 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.573937+0000", 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0059999999999998e-05 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2343 3.9 e33/21 3:954d6f9d:::rbd_data.115e5ff16b0c.0000000000000008:head v 33'27, mlcod=30'25)", 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.577760+0000", 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0066335769999999, 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091814499999999999, 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577760+0000", 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577760+0000", 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577761+0000", 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9500000000000003e-07 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577785+0000", 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4473000000000001e-05 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577798+0000", 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.279e-05 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577806+0000", 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3440000000000001e-06 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577864+0000", 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7357e-05 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577886+0000", 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2631999999999999e-05 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.578637+0000", 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075043699999999996 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.578678+0000", 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1616999999999998e-05 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2984 3.5 e33/21 3:a2b34f77:::rbd_data.115e5ff16b0c.000000000000001c:head v 33'37, mlcod=30'35)", 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.97088767099999995, 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437279999999999, 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613507+0000", 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.63e-07 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613521+0000", 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4545999999999999e-05 2024-04-24T00:34:05.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613540+0000", 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8963000000000001e-05 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613550+0000", 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0298e-05 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613622+0000", 2024-04-24T00:34:05.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1476999999999993e-05 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613645+0000", 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2940999999999999e-05 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.615112+0000", 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014666729999999999 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.615150+0000", 2024-04-24T00:34:05.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8367e-05 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2979 3.9 e33/21 3:9717bf35:::rbd_data.115e5ff16b0c.000000000000001b:head v 33'28, mlcod=30'26)", 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.76219232999999997, 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053184019999999999, 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822203+0000", 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3909999999999999e-06 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822482+0000", 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027925399999999999 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822509+0000", 2024-04-24T00:34:05.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.707e-05 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822521+0000", 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1779e-05 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822626+0000", 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105165 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822653+0000", 2024-04-24T00:34:05.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6713e-05 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827477+0000", 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048240949999999996 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827520+0000", 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2935000000000001e-05 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.1f epoch 33/21 rep_tid 217 entries 33'27 (0'0) error 3:fb1094b8:::rbd_header.115e5ff16b0c:head by client.4446.0:3392 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 32'25)", 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.74560283800000005, 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001188705, 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838792+0000", 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838804+0000", 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1861e-05 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838830+0000", 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5428000000000001e-05 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838845+0000", 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5729000000000001e-05 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838928+0000", 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2584999999999996e-05 2024-04-24T00:34:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.839980+0000", 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001051754 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3748 3.1f e33/21 3:feb5d058:::rbd_data.115e5ff16b0c.0000000000000030:head v 33'28, mlcod=33'26)", 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.70540412699999999, 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028352379999999999, 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878991+0000", 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879014+0000", 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2677e-05 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879033+0000", 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9361999999999999e-05 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879043+0000", 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2299999999999997e-06 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879110+0000", 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7385999999999995e-05 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879132+0000", 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2405e-05 2024-04-24T00:34:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881782+0000", 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026490979999999999 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881825+0000", 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3460000000000001e-05 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3747 3.9 e33/21 3:965825a6:::rbd_data.115e5ff16b0c.0000000000000038:head v 33'29, mlcod=33'27)", 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.69686667599999996, 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003408908, 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887528+0000", 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2240000000000001e-06 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887593+0000", 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4362999999999993e-05 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887606+0000", 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887614+0000", 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2290000000000003e-06 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887680+0000", 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5896999999999996e-05 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887704+0000", 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3516e-05 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890888+0000", 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031837860000000001 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890936+0000", 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8464000000000002e-05 2024-04-24T00:34:05.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3763 3.9 e33/21 3:97b67259:::rbd_data.115e5ff16b0c.0000000000000034:head v 33'30, mlcod=33'28)", 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.689739455, 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013435750000000001, 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894656+0000", 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.114e-06 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894684+0000", 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8742e-05 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894699+0000", 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5028000000000001e-05 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894708+0000", 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2260000000000005e-06 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894771+0000", 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3485000000000006e-05 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894792+0000", 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1393e-05 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.895944+0000", 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001151972 2024-04-24T00:34:05.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.895998+0000", 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3615000000000001e-05 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3899 3.1f e33/21 3:fb95f7ed:::rbd_data.115e5ff16b0c.000000000000003c:head v 33'29, mlcod=33'27)", 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.68615848400000001, 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001093927, 2024-04-24T00:34:05.657 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898237+0000", 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.145e-06 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898269+0000", 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540999999999999e-05 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898285+0000", 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5421e-05 2024-04-24T00:34:05.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898292+0000", 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0720000000000002e-06 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898352+0000", 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0182000000000001e-05 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898374+0000", 2024-04-24T00:34:05.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2716000000000001e-05 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.899272+0000", 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089749399999999998 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.899329+0000", 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355999999999998e-05 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:4343 3.4 e33/21 3:276ea59a:::rbd_data.115e5ff16b0c.0000000000000051:head v 33'14, mlcod=30'12)", 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.59828426400000001, 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012162010000000001, 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:05.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986111+0000", 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.336e-06 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986155+0000", 2024-04-24T00:34:05.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3664000000000001e-05 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986178+0000", 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3314000000000001e-05 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986188+0000", 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0485e-05 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986254+0000", 2024-04-24T00:34:05.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5117999999999996e-05 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986276+0000", 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2586e-05 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.987285+0000", 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001008695 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.987326+0000", 2024-04-24T00:34:05.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1003e-05 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:4408 3.1f e33/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 33'30, mlcod=33'28)", 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:05.000037+0000", 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.58435720599999996, 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065109300000000005, 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000037+0000", 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000037+0000", 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000038+0000", 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0869999999999999e-06 2024-04-24T00:34:05.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000045+0000", 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4290000000000004e-06 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000060+0000", 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4620999999999999e-05 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000080+0000", 2024-04-24T00:34:05.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0248999999999999e-05 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000135+0000", 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5182000000000001e-05 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000151+0000", 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5676e-05 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000637+0000", 2024-04-24T00:34:05.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048582600000000002 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.000688+0000", 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1023000000000003e-05 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:4426 3.1f e33/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 33'33, mlcod=33'31)", 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:05.011321+0000", 2024-04-24T00:34:05.670 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.57307302800000004, 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062697499999999999, 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011321+0000", 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011321+0000", 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:05.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011322+0000", 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4399999999999999e-07 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011329+0000", 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2779999999999996e-06 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011342+0000", 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2629000000000001e-05 2024-04-24T00:34:05.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011348+0000", 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8210000000000004e-06 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011398+0000", 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9805000000000002e-05 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011414+0000", 2024-04-24T00:34:05.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6344000000000001e-05 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011922+0000", 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050734599999999999 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.011948+0000", 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6007999999999999e-05 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:05.675 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:05.675 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:05.675 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:05.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:05.677 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:05.678 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:05.678 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:05.678 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:05.695 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:05.695 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:05.688+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:05.716 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:05.716 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:05.718 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-04-24T00:34:05.780 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:05.776+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:05.822 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:05.822 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:05.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:05.882 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:05.876+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:05.953 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:05.953 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:05.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:05.999 INFO:teuthology.orchestra.run.smithi064.stdout:523 punch from 0xaebdd74 to 0xaec2b5d, (0x4de9 bytes) 2024-04-24T00:34:05.999 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:05.992+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:06.002 INFO:teuthology.orchestra.run.smithi064.stdout:524 read 0x2d39cec thru 0x2d4329b (0x95b0 bytes) 2024-04-24T00:34:06.003 INFO:teuthology.orchestra.run.smithi064.stdout:526 writesame 0xd45b978 thru 0xd45e0da data_size 0x1(0x2763 bytes) 2024-04-24T00:34:06.050 INFO:teuthology.orchestra.run.smithi064.stdout:527 write 0xe798c8f thru 0xe7993f1 (0x763 bytes) 2024-04-24T00:34:06.056 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:06.056 INFO:teuthology.orchestra.run.smithi064.stdout:530 punch from 0x58b3dab to 0x58c25e4, (0xe839 bytes) 2024-04-24T00:34:06.056 INFO:teuthology.orchestra.run.smithi064.stdout:531 write 0xc9e8fd5 thru 0xc9f7b88 (0xebb4 bytes) 2024-04-24T00:34:06.067 INFO:teuthology.orchestra.run.smithi064.stdout:533 writesame 0x8192546 thru 0x819b142 data_size 0x1(0x8bfd bytes) 2024-04-24T00:34:06.084 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:06.080+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:06.104 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:06.104 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:06.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:06.154 INFO:teuthology.orchestra.run.smithi064.stdout:534 read 0x5601a5c thru 0x5607a6d (0x6012 bytes) 2024-04-24T00:34:06.155 INFO:teuthology.orchestra.run.smithi064.stdout:535 write 0x8be77ea thru 0x8bec468 (0x4c7f bytes) 2024-04-24T00:34:06.162 INFO:teuthology.orchestra.run.smithi064.stdout:536 write 0x2d0d49a thru 0x2d182bd (0xae24 bytes) 2024-04-24T00:34:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:537 write 0xad628db thru 0xad684eb (0x5c11 bytes) 2024-04-24T00:34:06.185 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:06.180+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:06.197 INFO:teuthology.orchestra.run.smithi064.stdout:538 trunc from 0xec0dc1f to 0x4d6a300 2024-04-24T00:34:06.244 INFO:teuthology.orchestra.run.smithi064.stdout:539 punch from 0x98623f to 0x98f6f5, (0x94b6 bytes) 2024-04-24T00:34:06.244 INFO:teuthology.orchestra.run.smithi064.stdout:540 punch from 0x42e642c to 0x42e8b87, (0x275b bytes) 2024-04-24T00:34:06.244 INFO:teuthology.orchestra.run.smithi064.stdout:541 write 0x1669cf thru 0x175e69 (0xf49b bytes) 2024-04-24T00:34:06.254 INFO:teuthology.orchestra.run.smithi064.stdout:542 write 0xd59fd9 thru 0xd693c0 (0xf3e8 bytes) 2024-04-24T00:34:06.260 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:06.260 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:06.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:06.281 INFO:teuthology.orchestra.run.smithi064.stdout:544 trunc from 0x4d6a300 to 0xc48fba7 2024-04-24T00:34:06.285 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:06.284+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:06.291 INFO:teuthology.orchestra.run.smithi064.stdout:547 read 0xae339af thru 0xae4370d (0xfd5f bytes) 2024-04-24T00:34:06.291 INFO:teuthology.orchestra.run.smithi064.stdout:549 read 0x542fb8a thru 0x543cc6e (0xd0e5 bytes) 2024-04-24T00:34:06.291 INFO:teuthology.orchestra.run.smithi064.stdout:550 punch from 0x876a1a8 to 0x8775509, (0xb361 bytes) 2024-04-24T00:34:06.308 INFO:teuthology.orchestra.run.smithi064.stdout:552 trunc from 0xc48fba7 to 0x49f8998 2024-04-24T00:34:06.336 INFO:teuthology.orchestra.run.smithi064.stdout:553 write 0x36c30d5 thru 0x36c6913 (0x383f bytes) 2024-04-24T00:34:06.339 INFO:teuthology.orchestra.run.smithi064.stdout:554 read 0x145b7b9 thru 0x1462b72 (0x73ba bytes) 2024-04-24T00:34:06.381 INFO:teuthology.orchestra.run.smithi064.stdout:557 write 0xed2297c thru 0xed2f38d (0xca12 bytes) 2024-04-24T00:34:06.386 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:06.384+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:06.390 INFO:teuthology.orchestra.run.smithi064.stdout:558 write 0x7f8d619 thru 0x7f944bf (0x6ea7 bytes) 2024-04-24T00:34:06.414 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:06.414 INFO:teuthology.orchestra.run.smithi064.stdout:561 read 0x33c21a0 thru 0x33d04b6 (0xe317 bytes) 2024-04-24T00:34:06.415 INFO:teuthology.orchestra.run.smithi064.stdout:563 writesame 0xc624786 thru 0xc62e08f data_size 0x13(0x990a bytes) 2024-04-24T00:34:06.430 INFO:teuthology.orchestra.run.smithi064.stdout:564 punch from 0x2110c20 to 0x211519d, (0x457d bytes) 2024-04-24T00:34:06.431 INFO:teuthology.orchestra.run.smithi064.stdout:566 read 0xc2cc0a1 thru 0xc2cc31d (0x27d bytes) 2024-04-24T00:34:06.431 INFO:teuthology.orchestra.run.smithi064.stdout:568 read 0xd138302 thru 0xd141809 (0x9508 bytes) 2024-04-24T00:34:06.431 INFO:teuthology.orchestra.run.smithi064.stdout:569 writesame 0x2c95ca1 thru 0x2c9d4b1 data_size 0x1(0x7811 bytes) 2024-04-24T00:34:06.447 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:06.447 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:06.462 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:06.511 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:06.508+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:06.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:34:06.551 INFO:teuthology.orchestra.run.smithi064.stdout:571 read 0x147189b thru 0x147e784 (0xceea bytes) 2024-04-24T00:34:06.552 INFO:teuthology.orchestra.run.smithi064.stdout:573 read 0x1d77d9c thru 0x1d82654 (0xa8b9 bytes) 2024-04-24T00:34:06.553 INFO:teuthology.orchestra.run.smithi064.stdout:574 read 0x999802a thru 0x99a01b1 (0x8188 bytes) 2024-04-24T00:34:06.553 INFO:teuthology.orchestra.run.smithi064.stdout:575 read 0xf8f725 thru 0xf98e16 (0x96f2 bytes) 2024-04-24T00:34:06.553 INFO:teuthology.orchestra.run.smithi064.stdout:577 writesame 0x38c24de thru 0x38d1cf9 data_size 0x4f(0xf81c bytes) 2024-04-24T00:34:06.556 INFO:teuthology.orchestra.run.smithi064.stdout:578 write 0x3e4878d thru 0x3e4beb1 (0x3725 bytes) 2024-04-24T00:34:06.562 INFO:teuthology.orchestra.run.smithi064.stdout:579 writesame 0x31e02e1 thru 0x31eec4e data_size 0x1(0xe96e bytes) 2024-04-24T00:34:06.586 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:06.586 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:06.588 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:06.588+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:06.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:06.689 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:06.684+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi064.stdout:580 read 0x8488e6c thru 0x848d094 (0x4229 bytes) 2024-04-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi064.stdout:583 writesame 0x7a29f52 thru 0x7a35f2b data_size 0xd(0xbfda bytes) 2024-04-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi064.stdout:584 read 0xdeb8c24 thru 0xdeb8cdd (0xba bytes) 2024-04-24T00:34:06.738 INFO:teuthology.orchestra.run.smithi064.stdout:585 trunc from 0xed2f38e to 0xbf4481 2024-04-24T00:34:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:587 read 0xaaf6c1 thru 0xab0cbf (0x15ff bytes) 2024-04-24T00:34:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:588 read 0x3c6e55 thru 0x3c7d46 (0xef2 bytes) 2024-04-24T00:34:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:589 read 0x148657 thru 0x14dd24 (0x56ce bytes) 2024-04-24T00:34:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:591 read 0xba80e7 thru 0xbaa576 (0x2490 bytes) 2024-04-24T00:34:06.769 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:06.769 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:06.776 INFO:teuthology.orchestra.run.smithi064.stdout:592 write 0xce303b thru 0xce4399 (0x135f bytes) 2024-04-24T00:34:06.779 INFO:teuthology.orchestra.run.smithi064.stdout:593 read 0x53837 thru 0x61dc9 (0xe593 bytes) 2024-04-24T00:34:06.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:06.794 INFO:teuthology.orchestra.run.smithi064.stdout:594 trunc from 0xce439a to 0xa133234 2024-04-24T00:34:06.801 INFO:teuthology.orchestra.run.smithi064.stdout:596 writesame 0x57b0c0 thru 0x5813c1 data_size 0x2e(0x6302 bytes) 2024-04-24T00:34:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:606 read 0x519e69e thru 0x51a8021 (0x9984 bytes) 2024-04-24T00:34:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:608 punch from 0x46ae925 to 0x46b168b, (0x2d66 bytes) 2024-04-24T00:34:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:609 write 0x701b4f thru 0x70d3b8 (0xb86a bytes) 2024-04-24T00:34:06.809 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:06.810 INFO:teuthology.orchestra.run.smithi064.stdout:612 write 0x9af894 thru 0x9b0f43 (0x16b0 bytes) 2024-04-24T00:34:06.812 INFO:teuthology.orchestra.run.smithi064.stdout:613 write 0x8a8d987 thru 0x8a9aec7 (0xd541 bytes) 2024-04-24T00:34:06.821 INFO:teuthology.orchestra.run.smithi064.stdout:615 write 0x27deb14 thru 0x27eb6ec (0xcbd9 bytes) 2024-04-24T00:34:06.827 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:06.824+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:06.828 INFO:teuthology.orchestra.run.smithi064.stdout:616 read 0x5cd830d thru 0x5ce72e7 (0xefdb bytes) 2024-04-24T00:34:06.828 INFO:teuthology.orchestra.run.smithi064.stdout:617 read 0x282c96a thru 0x28365e9 (0x9c80 bytes) 2024-04-24T00:34:06.828 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0xe8e14b1 2024-04-24T00:34:06.835 INFO:teuthology.orchestra.run.smithi064.stdout:618 trunc from 0xa133234 to 0xe8e14b1 2024-04-24T00:34:06.844 INFO:teuthology.orchestra.run.smithi064.stdout:619 write 0x9157e3a thru 0x9162cd0 (0xae97 bytes) 2024-04-24T00:34:06.851 INFO:teuthology.orchestra.run.smithi064.stdout:622 write 0xc6dd2df thru 0xc6e54aa (0x81cc bytes) 2024-04-24T00:34:06.858 INFO:teuthology.orchestra.run.smithi064.stdout:624 clone 3 order 20 su 262144 sc 12 2024-04-24T00:34:06.891 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:06.888+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:06.953 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:06.953 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:06.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:07.011 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:07.008+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:07.093 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:07.092+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:07.129 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:07.129 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:07.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:07.194 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:07.188+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:07.267 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:07.267 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:07.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:07.295 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:07.292+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:07.396 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:07.392+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:07.445 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:07.445 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:07.458 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:07.497 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:07.496+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:07.556 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:34:07.598 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:07.592+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:07.625 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:07.626 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:07.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:07.698 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:07.696+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:07.737 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:07.799 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:07.796+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:07.809 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:07.809 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:07.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:07.896 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:07.897 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:07.899 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:07.896+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:07.913 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:07.930 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:07.930 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:07.940 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:07.940 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:07.946 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:07.946 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:07.954 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:07.964 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:07.964 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:07.983 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:07.983 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:07.999 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:07.996+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:08.003 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:08.003 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:08.022 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:08.022 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:08.101 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:08.096+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:08.121 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.189429581999999, 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:34:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:34:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.735968891999999, 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019315533999999999, 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 112 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379866+0000", 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.379868+0000", 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.274e-06 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380096+0000", 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022802799999999999 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380146+0000", 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0358999999999997e-05 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380164+0000", 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7713000000000001e-05 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391250+0000", 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011085928 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391383+0000", 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013310899999999999 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.391576+0000", 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019287899999999999 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395703+0000", 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041278740000000001 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398947+0000", 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032432149999999998 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399035+0000", 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8635999999999996e-05 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399182+0000", 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014651900000000001 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.735547005000001, 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019317091000000002, 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 114 2024-04-24T00:34:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380288+0000", 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380289+0000", 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.173e-06 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380300+0000", 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-05 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380324+0000", 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3047000000000001e-05 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380338+0000", 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4545999999999999e-05 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394634+0000", 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014296233 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394760+0000", 2024-04-24T00:34:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012605900000000001 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.394953+0000", 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019237900000000001 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398924+0000", 2024-04-24T00:34:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039709589999999996 2024-04-24T00:34:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399439+0000", 2024-04-24T00:34:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051536599999999996 2024-04-24T00:34:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399507+0000", 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8065999999999995e-05 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399605+0000", 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8133000000000004e-05 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.735433542999999, 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020360119999999999, 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 115 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380402+0000", 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380404+0000", 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:34:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380546+0000", 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014221200000000001 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380561+0000", 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5696e-05 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380572+0000", 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0353999999999999e-05 2024-04-24T00:34:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395831+0000", 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015259435 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.395982+0000", 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150736 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.396234+0000", 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000252334 2024-04-24T00:34:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399712+0000", 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034776529999999998 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400554+0000", 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00084231699999999998 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400634+0000", 2024-04-24T00:34:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9270000000000005e-05 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400762+0000", 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012814900000000001 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.735243706999999, 2024-04-24T00:34:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020559280999999999, 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 116 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380591+0000", 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380593+0000", 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380600+0000", 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0160000000000003e-06 2024-04-24T00:34:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380614+0000", 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4317e-05 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380622+0000", 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.397613+0000", 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016991532 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.397762+0000", 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149228 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.398030+0000", 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026737499999999998 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400774+0000", 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027444959999999999 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400968+0000", 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193358 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401045+0000", 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7440000000000004e-05 2024-04-24T00:34:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401151+0000", 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105719 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.735200848000002, 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024520601, 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 117 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380635+0000", 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380642+0000", 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7700000000000004e-06 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380654+0000", 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1686e-05 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380666+0000", 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2150000000000001e-05 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399787+0000", 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019121461999999999 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399921+0000", 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134043 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400127+0000", 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020607500000000001 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401186+0000", 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001058738 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.404952+0000", 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037661779999999998 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405042+0000", 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9889999999999995e-05 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405155+0000", 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112812 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:2112 3.15 3:ae8ff710:::rbd_object_map.115e5ff16b0c.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.540193+0000", 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5756421180000002, 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022449243000000001, 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 2112 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.540193+0000", 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.540193+0000", 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.540196+0000", 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4839999999999998e-06 2024-04-24T00:34:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.540208+0000", 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2408999999999999e-05 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.540238+0000", 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.974e-05 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.540259+0000", 2024-04-24T00:34:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1693000000000001e-05 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.540438+0000", 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017871900000000001 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.540463+0000", 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.459e-05 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.547874+0000", 2024-04-24T00:34:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074110060000000004 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.548134+0000", 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026080100000000001 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.549318+0000", 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011834510000000001 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.554359+0000", 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005040704 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.562488+0000", 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0081298039999999992 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.562572+0000", 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3209999999999998e-05 2024-04-24T00:34:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.562642+0000", 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0631999999999997e-05 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:2113 3.15 3:ae8ff710:::rbd_object_map.115e5ff16b0c.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.543418+0000", 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.572417186, 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019377950000000001, 2024-04-24T00:34:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 2113 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.543418+0000", 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.543418+0000", 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.543421+0000", 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8559999999999998e-06 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.543436+0000", 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5497000000000001e-05 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.543471+0000", 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5006000000000003e-05 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.543532+0000", 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0553000000000001e-05 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.543751+0000", 2024-04-24T00:34:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021878500000000001 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.543786+0000", 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4990000000000002e-05 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.550603+0000", 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068169040000000004 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.550806+0000", 2024-04-24T00:34:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000203809 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.551711+0000", 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00090486600000000005 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.554466+0000", 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027542619999999999 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.562681+0000", 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082151399999999992 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.562747+0000", 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5902999999999996e-05 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.562796+0000", 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9379000000000003e-05 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3883 3.11 3:8c019bdf:::rbd_data.115eb9c76af6.0000000000000007:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.892167+0000", 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.2236680899999999, 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019371165999999999, 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3883 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892167+0000", 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892167+0000", 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892168+0000", 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2500000000000005e-07 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892170+0000", 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1730000000000002e-06 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892178+0000", 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2539999999999992e-06 2024-04-24T00:34:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892197+0000", 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8518000000000001e-05 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.911257+0000", 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019060187999999999 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.911384+0000", 2024-04-24T00:34:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012693900000000001 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.911538+0000", 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015446899999999999 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3884 3.11 3:8c019bdf:::rbd_data.115eb9c76af6.0000000000000007:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.892205+0000", 2024-04-24T00:34:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.2236300899999999, 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019761399999999998, 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3884 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892205+0000", 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892205+0000", 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892206+0000", 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9999999999999997e-07 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892217+0000", 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1044e-05 2024-04-24T00:34:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892224+0000", 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4880000000000001e-06 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892230+0000", 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6980000000000003e-06 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.911618+0000", 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019387596 2024-04-24T00:34:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.911794+0000", 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017621600000000001 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.911966+0000", 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017275799999999999 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3885 3.11 3:8c019bdf:::rbd_data.115eb9c76af6.0000000000000007:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.892237+0000", 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.223597743, 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020004669999999999, 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3885 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892237+0000", 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892237+0000", 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892238+0000", 2024-04-24T00:34:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2300000000000001e-07 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892240+0000", 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3719999999999999e-06 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892247+0000", 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.455e-06 2024-04-24T00:34:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.892255+0000", 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1370000000000002e-06 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.912010+0000", 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019755002000000001 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.912099+0000", 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8667999999999999e-05 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.912242+0000", 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143413 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4073 3.11 3:8c019bdf:::rbd_data.115eb9c76af6.0000000000000007:5 [sparse-read 5636096~0 [fadvise_nocache]] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.937273+0000", 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.1785623429999998, 2024-04-24T00:34:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019593892000000002, 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4073 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937273+0000", 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937273+0000", 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937273+0000", 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2399999999999998e-07 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937287+0000", 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3076e-05 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937300+0000", 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3209e-05 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937306+0000", 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6900000000000003e-06 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.956617+0000", 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019310708999999999 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.956737+0000", 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120345 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.956867+0000", 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012923899999999999 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4074 3.11 3:8c019bdf:::rbd_data.115eb9c76af6.0000000000000007:5 [sparse-read 5767168~0 [fadvise_nocache]] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.937345+0000", 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.1784898030000002, 2024-04-24T00:34:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019952668999999999, 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4074 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937345+0000", 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937345+0000", 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937347+0000", 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3769999999999999e-06 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937351+0000", 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6870000000000002e-06 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937365+0000", 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4048e-05 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937378+0000", 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2771e-05 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.956942+0000", 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019563727 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.957090+0000", 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014848500000000001 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.957298+0000", 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000207574 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4075 3.11 3:8c019bdf:::rbd_data.115eb9c76af6.0000000000000007:5 [sparse-read 5898240~0 [fadvise_nocache]] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.937394+0000", 2024-04-24T00:34:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.1784411289999999, 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020255809, 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4075 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937394+0000", 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937394+0000", 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937395+0000", 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937408+0000", 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2829e-05 2024-04-24T00:34:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937424+0000", 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6223e-05 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937431+0000", 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7309999999999996e-06 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.957377+0000", 2024-04-24T00:34:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019945912999999999 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.957504+0000", 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012671899999999999 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.957650+0000", 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014613399999999999 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4076 3.11 3:8c019bdf:::rbd_data.115eb9c76af6.0000000000000007:5 [sparse-read 6029312~0 [fadvise_nocache]] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.937441+0000", 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.1783942769999998, 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020645257, 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4076 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937441+0000", 2024-04-24T00:34:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937441+0000", 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937442+0000", 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.92e-07 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937444+0000", 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7949999999999999e-06 2024-04-24T00:34:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937463+0000", 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8198000000000001e-05 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937479+0000", 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5991999999999999e-05 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.957733+0000", 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020253990999999999 2024-04-24T00:34:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.957896+0000", 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000163265 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958086+0000", 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019032399999999999 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4077 3.11 3:8c019bdf:::rbd_data.115eb9c76af6.0000000000000007:5 [sparse-read 6160384~0 [fadvise_nocache]] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.937525+0000", 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.178310492, 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020864146, 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4077 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937525+0000", 2024-04-24T00:34:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937525+0000", 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937526+0000", 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.277e-06 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937531+0000", 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2850000000000002e-06 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937548+0000", 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6310000000000001e-05 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937571+0000", 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3572000000000001e-05 2024-04-24T00:34:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958150+0000", 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020578486999999999 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958251+0000", 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010116900000000001 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958389+0000", 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013804600000000001 2024-04-24T00:34:08.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4080 3.11 3:8da71efb:::rbd_data.115eb9c76af6.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.937945+0000", 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.177889961, 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020820768, 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4080 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937945+0000", 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937945+0000", 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937947+0000", 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.668e-06 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937954+0000", 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2019999999999998e-06 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.937968+0000", 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4382000000000001e-05 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938008+0000", 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9245000000000001e-05 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958467+0000", 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020458929000000001 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958628+0000", 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161371 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958766+0000", 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000137971 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4081 3.11 3:8d5cda87:::rbd_data.115eb9c76af6.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.938053+0000", 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.177781864, 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020999653, 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4081 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938053+0000", 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938053+0000", 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938054+0000", 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3900000000000003e-07 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938058+0000", 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8829999999999999e-06 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938073+0000", 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4551e-05 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938080+0000", 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6190000000000002e-06 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958835+0000", 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020755136 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.958927+0000", 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1918999999999999e-05 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.959053+0000", 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125606 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.216 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4084 3.11 3:8da71efb:::rbd_data.115eb9c76af6.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.938107+0000", 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.1777281569999998, 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021322266999999999, 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4084 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938107+0000", 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938107+0000", 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938108+0000", 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6500000000000008e-07 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938112+0000", 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0740000000000003e-06 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938128+0000", 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6273000000000002e-05 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938138+0000", 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8360000000000004e-06 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.959157+0000", 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021018411000000001 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.959281+0000", 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012401299999999999 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.959429+0000", 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014869500000000001 2024-04-24T00:34:08.220 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4085 3.11 3:8d5cda87:::rbd_data.115eb9c76af6.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.938152+0000", 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.1776831099999998, 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021540894000000001, 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:08.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4085 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938152+0000", 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:08.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938152+0000", 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938153+0000", 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7999999999999995e-07 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938156+0000", 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.802e-06 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:08.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938166+0000", 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9660000000000008e-06 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.938171+0000", 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8340000000000004e-06 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.959504+0000", 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021332365999999998 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:08.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.959593+0000", 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9430000000000003e-05 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.959693+0000", 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9815999999999994e-05 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:08.225 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:08.227 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:08.224+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:08.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:08.327 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:08.324+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:08.362 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:08.362 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:08.362 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:08.362 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:08.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:08.428 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:08.424+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:08.497 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:08.497 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:08.497 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:08.497 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:08.497 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:08.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:08.530 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:08.524+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:08.630 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:08.624+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:08.664 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:08.664 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:08.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:08.664 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:140 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'85)", 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.764033210000001, 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012183782000000001, 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897149+0000", 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.147e-06 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897223+0000", 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4116000000000007e-05 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897251+0000", 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7606e-05 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897269+0000", 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7476999999999999e-05 2024-04-24T00:34:08.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902553+0000", 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052847359999999999 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902586+0000", 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2802000000000002e-05 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909235+0000", 2024-04-24T00:34:08.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066493519999999999 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909332+0000", 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6545999999999993e-05 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'85)", 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.763223100000001, 2024-04-24T00:34:08.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011427935, 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897960+0000", 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1500000000000002e-06 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897974+0000", 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3890000000000001e-05 2024-04-24T00:34:08.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898000+0000", 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.599e-05 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898018+0000", 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7903000000000001e-05 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903913+0000", 2024-04-24T00:34:08.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058942839999999996 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903976+0000", 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3847e-05 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909338+0000", 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053611850000000001 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909386+0000", 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8686000000000001e-05 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1964 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 983040~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530433+0000", 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1307483789999999, 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011371305, 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1964 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530433+0000", 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530433+0000", 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530433+0000", 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4799999999999999e-07 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0090000000000004e-06 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530446+0000", 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1929999999999998e-06 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530450+0000", 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7840000000000003e-06 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541633+0000", 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011182588 2024-04-24T00:34:08.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541705+0000", 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2416999999999995e-05 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8865999999999998e-05 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.679 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1981 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.679 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530458+0000", 2024-04-24T00:34:08.679 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1307238310000001, 2024-04-24T00:34:08.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011545852000000001, 2024-04-24T00:34:08.679 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1981 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530458+0000", 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530458+0000", 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530458+0000", 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0399999999999996e-07 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530461+0000", 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9840000000000001e-06 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530467+0000", 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1859999999999997e-06 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530480+0000", 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2335e-05 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541833+0000", 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011352948999999999 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541906+0000", 2024-04-24T00:34:08.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3385000000000003e-05 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542003+0000", 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7509000000000004e-05 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1982 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530487+0000", 2024-04-24T00:34:08.684 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1306940030000003, 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012076564, 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1982 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530487+0000", 2024-04-24T00:34:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530487+0000", 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530488+0000", 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530490+0000", 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3609999999999999e-06 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530497+0000", 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3820000000000001e-06 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530501+0000", 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542356+0000", 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011854131 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542441+0000", 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5604000000000002e-05 2024-04-24T00:34:08.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542564+0000", 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012279599999999999 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1983 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1310720~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530508+0000", 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1306732750000004, 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012285136, 2024-04-24T00:34:08.689 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1983 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530508+0000", 2024-04-24T00:34:08.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530508+0000", 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530509+0000", 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530511+0000", 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1280000000000002e-06 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530517+0000", 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0410000000000003e-06 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530521+0000", 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4179999999999999e-06 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542595+0000", 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012073486 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542668+0000", 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3126000000000004e-05 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542793+0000", 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000125573 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1984 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1441792~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530527+0000", 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1306542840000002, 2024-04-24T00:34:08.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012414612, 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1984 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530527+0000", 2024-04-24T00:34:08.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530527+0000", 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530528+0000", 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899999999999998e-07 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530530+0000", 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0150000000000002e-06 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530536+0000", 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1999999999999999e-06 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530540+0000", 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4730000000000002e-06 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542825+0000", 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012285007000000001 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542880+0000", 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4598999999999997e-05 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542942+0000", 2024-04-24T00:34:08.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1948999999999999e-05 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1985 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1572864~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1306349300000003, 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012525493, 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.699 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1985 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:08.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530547+0000", 2024-04-24T00:34:08.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:34:08.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530549+0000", 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.08e-06 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530555+0000", 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7980000000000002e-06 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530559+0000", 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7380000000000002e-06 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542973+0000", 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01241387 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543014+0000", 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0392000000000002e-05 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543072+0000", 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8248999999999997e-05 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1986 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1703936~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530567+0000", 2024-04-24T00:34:08.704 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1306148910000005, 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012628393999999999, 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1986 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530567+0000", 2024-04-24T00:34:08.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530567+0000", 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530567+0000", 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6199999999999999e-07 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530569+0000", 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1349999999999999e-06 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530575+0000", 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1489999999999999e-06 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530582+0000", 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543099+0000", 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012517014 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543140+0000", 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0951000000000001e-05 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543195+0000", 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5106999999999999e-05 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.708 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1987 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1835008~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1305925300000004, 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012737480000000001, 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.709 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1987 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899999999999998e-07 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530598+0000", 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3909999999999998e-06 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530605+0000", 2024-04-24T00:34:08.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4290000000000004e-06 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530610+0000", 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0080000000000002e-06 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543224+0000", 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012613928999999999 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543271+0000", 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7333999999999998e-05 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543326+0000", 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5019999999999998e-05 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1988 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1966080~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530620+0000", 2024-04-24T00:34:08.713 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1305617769999996, 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012851636, 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1988 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530620+0000", 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530620+0000", 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530620+0000", 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530623+0000", 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2929999999999999e-06 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530629+0000", 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.28e-06 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9699999999999994e-06 2024-04-24T00:34:08.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543354+0000", 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012715371 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543394+0000", 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9444999999999999e-05 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543471+0000", 2024-04-24T00:34:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7700999999999993e-05 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2013 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1114112~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530646+0000", 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.130535428, 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012949534, 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.718 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2013 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530646+0000", 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530646+0000", 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530646+0000", 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8099999999999998e-07 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530649+0000", 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.272e-06 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530655+0000", 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4540000000000004e-06 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530660+0000", 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543500+0000", 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012840014 2024-04-24T00:34:08.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543539+0000", 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8652000000000002e-05 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543596+0000", 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6777000000000002e-05 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2014 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1245184~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530667+0000", 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1305145339999996, 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013051086, 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2014 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530667+0000", 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530667+0000", 2024-04-24T00:34:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530667+0000", 2024-04-24T00:34:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2899999999999999e-07 2024-04-24T00:34:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530669+0000", 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530676+0000", 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1650000000000003e-06 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530680+0000", 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4719999999999997e-06 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543623+0000", 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012943378 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543662+0000", 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8058000000000001e-05 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543718+0000", 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6431000000000002e-05 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2015 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1376256~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1304950119999999, 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013207162, 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2015 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530687+0000", 2024-04-24T00:34:08.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8799999999999998e-07 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.001e-06 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530695+0000", 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2589999999999997e-06 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530700+0000", 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5310000000000003e-06 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543744+0000", 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013044616 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543795+0000", 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1084000000000001e-05 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543894+0000", 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8282999999999994e-05 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2016 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1507328~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530707+0000", 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1304740349999998, 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013377231999999999, 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2016 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530707+0000", 2024-04-24T00:34:08.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530707+0000", 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530708+0000", 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530714+0000", 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0589999999999999e-06 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530720+0000", 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2550000000000003e-06 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530725+0000", 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6190000000000002e-06 2024-04-24T00:34:08.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543923+0000", 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0131988 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543995+0000", 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1420000000000004e-05 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544085+0000", 2024-04-24T00:34:08.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9711999999999998e-05 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2017 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1638400~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530731+0000", 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1304502049999998, 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013532136, 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2017 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530731+0000", 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530731+0000", 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530732+0000", 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1300000000000001e-07 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530739+0000", 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6070000000000002e-06 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530746+0000", 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9009999999999997e-06 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530751+0000", 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0810000000000001e-06 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544115+0000", 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013363284 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544167+0000", 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2419000000000001e-05 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544263+0000", 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6430999999999999e-05 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2018 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1769472~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.742 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1304232409999999, 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013700711000000001, 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2018 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530759+0000", 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7099999999999997e-07 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530761+0000", 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1550000000000001e-06 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530767+0000", 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1639999999999998e-06 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530772+0000", 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.848e-06 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544294+0000", 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013521948000000001 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544368+0000", 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4207000000000003e-05 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544459+0000", 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1018000000000002e-05 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2019 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1304030709999999, 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014241181, 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.747 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2019 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:08.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530779+0000", 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530780+0000", 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.686e-06 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.507e-06 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530790+0000", 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5240000000000002e-06 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544487+0000", 2024-04-24T00:34:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013696612 2024-04-24T00:34:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544570+0000", 2024-04-24T00:34:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2704e-05 2024-04-24T00:34:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545020+0000", 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044978100000000001 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2020 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1303844859999996, 2024-04-24T00:34:08.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014215145, 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2020 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9499999999999998e-07 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530799+0000", 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.013e-06 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4289999999999998e-06 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530810+0000", 2024-04-24T00:34:08.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4719999999999997e-06 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544812+0000", 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01400174 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544855+0000", 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3217999999999999e-05 2024-04-24T00:34:08.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545012+0000", 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000156878 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:3707 3.1 3:860763fd:::rbd_data.115e5ff16b0c.000000000000002d:head [zero 0~141397,write 141397~52132 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.869574+0000", 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7916078720000002, 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012924814, 2024-04-24T00:34:08.757 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3707 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869574+0000", 2024-04-24T00:34:08.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869574+0000", 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869575+0000", 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7499999999999998e-07 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869625+0000", 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0260000000000003e-05 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869638+0000", 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3706999999999999e-05 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869658+0000", 2024-04-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869731+0000", 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2564999999999994e-05 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869886+0000", 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582399999999999 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.870086+0000", 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000199507 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.874167+0000", 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040809339999999996 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:08.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.882342+0000", 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081753469999999995 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.882403+0000", 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0705999999999998e-05 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.882498+0000", 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5451999999999993e-05 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:08.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:08.766 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:08.760+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:08.866 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:08.864+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:08.897 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:08.897 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:08.897 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:08.897 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:08.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:08.966 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:08.960+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:09.067 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:09.064+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:09.077 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:09.077 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:09.078 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:09.078 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:09.078 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:09.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:09.168 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:09.164+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:09.217 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:09.217 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:09.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2025 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1638400~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6853974560000005, 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077913280000000001, 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2025 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530291+0000", 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3200000000000005e-07 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530294+0000", 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530303+0000", 2024-04-24T00:34:09.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8820000000000007e-06 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530310+0000", 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0199999999999997e-06 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.537775+0000", 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074650569999999998 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.537853+0000", 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7961999999999997e-05 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538082+0000", 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0002288 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2026 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1769472~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.685368296, 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081592609999999993, 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2026 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530320+0000", 2024-04-24T00:34:09.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6700000000000003e-07 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530323+0000", 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7420000000000002e-06 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530332+0000", 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6059999999999992e-06 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530340+0000", 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000001e-06 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538180+0000", 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078402369999999999 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538308+0000", 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012834000000000001 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538479+0000", 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170507 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2027 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6853383309999996, 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.009211136, 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2027 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530350+0000", 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7199999999999999e-07 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530353+0000", 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3390000000000001e-06 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530362+0000", 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0489999999999993e-06 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530370+0000", 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5739999999999998e-06 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539401+0000", 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090311609999999994 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539466+0000", 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4880000000000002e-05 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539561+0000", 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4561000000000005e-05 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2028 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6853078080000001, 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095022319999999993, 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2028 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7899999999999998e-07 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530384+0000", 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0280000000000001e-06 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530392+0000", 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7830000000000004e-06 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530400+0000", 2024-04-24T00:34:09.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2080000000000003e-06 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539616+0000", 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092167310000000006 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539722+0000", 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010527900000000001 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539882+0000", 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000160624 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2045 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.235 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6852777450000005, 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097538699999999996, 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2045 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:09.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530411+0000", 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530414+0000", 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2430000000000001e-06 2024-04-24T00:34:09.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530423+0000", 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3030000000000005e-06 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530430+0000", 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.917e-06 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539923+0000", 2024-04-24T00:34:09.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094929349999999992 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540027+0000", 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010355 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540164+0000", 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013730799999999999 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.239 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2046 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6852475389999997, 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010030647, 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2046 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:09.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530441+0000", 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3200000000000005e-07 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530444+0000", 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8109999999999999e-06 2024-04-24T00:34:09.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530453+0000", 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3409999999999992e-06 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530460+0000", 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3719999999999999e-06 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540209+0000", 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097491539999999995 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540326+0000", 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001166 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540471+0000", 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014483700000000001 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2047 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:09.243 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6852168699999996, 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010295987, 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2047 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530472+0000", 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2799999999999995e-07 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530475+0000", 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0939999999999999e-06 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530485+0000", 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7969999999999995e-06 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530491+0000", 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8709999999999999e-06 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540514+0000", 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010022499000000001 2024-04-24T00:34:09.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540626+0000", 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111981 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540767+0000", 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014091700000000001 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2048 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6851860590000003, 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010591068, 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2048 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2699999999999999e-07 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530520+0000", 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8165999999999999e-05 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530533+0000", 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2065999999999999e-05 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:09.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540812+0000", 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010266239 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540925+0000", 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112676 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541093+0000", 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016786500000000001 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2049 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6851298159999999, 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010815047, 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2049 2024-04-24T00:34:09.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530562+0000", 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4369999999999999e-06 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530572+0000", 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530579+0000", 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541138+0000", 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010558209000000001 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541239+0000", 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101483 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541373+0000", 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133769 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2050 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6850984669999995, 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011052364, 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2050 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530590+0000", 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6599999999999996e-07 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530593+0000", 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.937e-06 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530602+0000", 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4360000000000008e-06 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530609+0000", 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0090000000000001e-06 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541411+0000", 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01080191 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541511+0000", 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9839000000000004e-05 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541642+0000", 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130667 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2051 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6850691070000003, 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011318002000000001, 2024-04-24T00:34:09.260 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2051 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2900000000000004e-07 2024-04-24T00:34:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530622+0000", 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9490000000000001e-06 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530632+0000", 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999994e-06 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.131e-06 2024-04-24T00:34:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541683+0000", 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011044629 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120383 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541937+0000", 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132941 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2052 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6850399070000002, 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011583872, 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2052 2024-04-24T00:34:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.44e-07 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530658+0000", 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9850000000000001e-06 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530668+0000", 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530675+0000", 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6240000000000002e-06 2024-04-24T00:34:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541978+0000", 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011302921 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542100+0000", 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121593 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542232+0000", 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013176600000000001 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2077 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6850025070000001, 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011842221, 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2077 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6000000000000003e-07 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.095e-06 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530699+0000", 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530706+0000", 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2429999999999999e-06 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542275+0000", 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011568746 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542386+0000", 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011176200000000001 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542527+0000", 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014114499999999999 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2078 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.684972181, 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012081010999999999, 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2078 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2200000000000004e-07 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530719+0000", 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9349999999999999e-06 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530728+0000", 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0860000000000008e-06 2024-04-24T00:34:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530735+0000", 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2169999999999997e-06 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542568+0000", 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011832540000000001 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542667+0000", 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9000999999999996e-05 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542797+0000", 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012971 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2079 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6849428680000003, 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012334128999999999, 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2079 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530748+0000", 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9040000000000001e-06 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2159999999999995e-06 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530764+0000", 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9890000000000003e-06 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542839+0000", 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012074354000000001 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542945+0000", 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106443 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543079+0000", 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013369599999999999 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2080 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6849140680000003, 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012588825999999999, 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2080 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530777+0000", 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.022e-06 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9879999999999993e-06 2024-04-24T00:34:09.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530793+0000", 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1409999999999999e-06 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543122+0000", 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012328274 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543230+0000", 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108046 2024-04-24T00:34:09.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543362+0000", 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013286799999999999 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2081 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6848851830000005, 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012843515, 2024-04-24T00:34:09.285 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2081 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1500000000000005e-07 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.819e-06 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530815+0000", 2024-04-24T00:34:09.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3540000000000006e-06 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530835+0000", 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9983000000000002e-05 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543405+0000", 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012569739 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543513+0000", 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010821500000000001 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543646+0000", 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289000000000001 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2082 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6848419679999997, 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013081859, 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2082 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530849+0000", 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0750000000000002e-06 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530859+0000", 2024-04-24T00:34:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7079999999999992e-06 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530866+0000", 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4309999999999997e-06 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543689+0000", 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012822750000000001 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543796+0000", 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106526 2024-04-24T00:34:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543928+0000", 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131846 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2083 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6848116950000005, 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013453964000000001, 2024-04-24T00:34:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2083 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530877+0000", 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530887+0000", 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186e-05 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530896+0000", 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5480000000000007e-06 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530904+0000", 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4599999999999997e-06 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543970+0000", 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013066067000000001 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544178+0000", 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020785000000000001 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544330+0000", 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015230699999999999 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2084 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.684773871, 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907047000000001, 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:09.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2084 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3099999999999998e-07 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530922+0000", 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.498e-06 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530932+0000", 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530940+0000", 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4769999999999997e-06 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544421+0000", 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013481739 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544673+0000", 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025143600000000001 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544821+0000", 2024-04-24T00:34:09.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014819 2024-04-24T00:34:09.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.302 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:09.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:09.304 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:09.300+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:09.404 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:09.404+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:09.476 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:09.476 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:09.476 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:09.476 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:09.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:09.533 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:09.532+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:09.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:34:09.606 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:09.600+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:09.672 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:09.672 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:09.672 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:09.672 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:09.672 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:09.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:09.707 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:09.704+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:09.782 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-04-24T00:34:09.808 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:09.804+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:09.820 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:11 3.1f e24/21 3:fa654f18:::rbd_id.image_client.0:head v 24'1, mlcod=0'0)", 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.779983696, 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001106349, 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.06e-07 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037959+0000", 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4007e-05 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037974+0000", 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4586999999999999e-05 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037985+0000", 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.137e-05 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.038111+0000", 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012570699999999999 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.038143+0000", 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2218999999999998e-05 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039003+0000", 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085975500000000002 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039051+0000", 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8097999999999999e-05 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1717 3.0 e31/21 3:04479396:::rbd_id.image_client.0-clone2:head v 31'1, mlcod=0'0)", 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:03.517225+0000", 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.3007033549999996, 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001023382, 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.825 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517225+0000", 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517225+0000", 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517225+0000", 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7000000000000001e-07 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517231+0000", 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5790000000000003e-06 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517246+0000", 2024-04-24T00:34:09.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5204e-05 2024-04-24T00:34:09.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517258+0000", 2024-04-24T00:34:09.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1552000000000001e-05 2024-04-24T00:34:09.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517322+0000", 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4103000000000006e-05 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.517345+0000", 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3212000000000001e-05 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.518195+0000", 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085033099999999996 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:03.518249+0000", 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3130999999999997e-05 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:1741 3.1f e32/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 32'24, mlcod=31'22)", 2024-04-24T00:34:09.830 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.501543+0000", 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.3163852379999996, 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083306799999999996, 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501543+0000", 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501543+0000", 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501544+0000", 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4799999999999998e-07 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501553+0000", 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0219999999999993e-06 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501582+0000", 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501600+0000", 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8363999999999999e-05 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501681+0000", 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0863000000000002e-05 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.501705+0000", 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4060999999999999e-05 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.502329+0000", 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062386799999999997 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.502376+0000", 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7172000000000002e-05 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2348 3.1f e33/21 3:fac32ee7:::rbd_data.115e5ff16b0c.0000000000000009:head v 33'26, mlcod=32'24)", 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2451412020000001, 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011498890000000001, 2024-04-24T00:34:09.834 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572788+0000", 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572816+0000", 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8061000000000001e-05 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572827+0000", 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1423999999999999e-05 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572835+0000", 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.858e-06 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572898+0000", 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3139000000000007e-05 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572929+0000", 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0355999999999999e-05 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.573887+0000", 2024-04-24T00:34:09.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095867599999999999 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.573937+0000", 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0059999999999998e-05 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2343 3.9 e33/21 3:954d6f9d:::rbd_data.115e5ff16b0c.0000000000000008:head v 33'27, mlcod=30'25)", 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.577760+0000", 2024-04-24T00:34:09.838 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.240168272, 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091814499999999999, 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577760+0000", 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577760+0000", 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577761+0000", 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9500000000000003e-07 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577785+0000", 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4473000000000001e-05 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577798+0000", 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.279e-05 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577806+0000", 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3440000000000001e-06 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577864+0000", 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7357e-05 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.577886+0000", 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2631999999999999e-05 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.578637+0000", 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075043699999999996 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.578678+0000", 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1616999999999998e-05 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2984 3.5 e33/21 3:a2b34f77:::rbd_data.115e5ff16b0c.000000000000001c:head v 33'37, mlcod=30'35)", 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2044223660000002, 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437279999999999, 2024-04-24T00:34:09.842 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613507+0000", 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.63e-07 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613521+0000", 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4545999999999999e-05 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613540+0000", 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8963000000000001e-05 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613550+0000", 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0298e-05 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613622+0000", 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1476999999999993e-05 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613645+0000", 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2940999999999999e-05 2024-04-24T00:34:09.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.615112+0000", 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014666729999999999 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.615150+0000", 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8367e-05 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2979 3.9 e33/21 3:9717bf35:::rbd_data.115e5ff16b0c.000000000000001b:head v 33'28, mlcod=30'26)", 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9957270249999999, 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053184019999999999, 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822203+0000", 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3909999999999999e-06 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822482+0000", 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027925399999999999 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822509+0000", 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.707e-05 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822521+0000", 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1779e-05 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822626+0000", 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105165 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822653+0000", 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6713e-05 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827477+0000", 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048240949999999996 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827520+0000", 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2935000000000001e-05 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.1f epoch 33/21 rep_tid 217 entries 33'27 (0'0) error 3:fb1094b8:::rbd_header.115e5ff16b0c:head by client.4446.0:3392 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 32'25)", 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:09.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9791375330000003, 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001188705, 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838792+0000", 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838804+0000", 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1861e-05 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838830+0000", 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5428000000000001e-05 2024-04-24T00:34:09.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838845+0000", 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5729000000000001e-05 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838928+0000", 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2584999999999996e-05 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.839980+0000", 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001051754 2024-04-24T00:34:09.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3748 3.1f e33/21 3:feb5d058:::rbd_data.115e5ff16b0c.0000000000000030:head v 33'28, mlcod=33'26)", 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9389388219999999, 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028352379999999999, 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878991+0000", 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879014+0000", 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2677e-05 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879033+0000", 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9361999999999999e-05 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879043+0000", 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2299999999999997e-06 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879110+0000", 2024-04-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7385999999999995e-05 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879132+0000", 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2405e-05 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881782+0000", 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026490979999999999 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881825+0000", 2024-04-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3460000000000001e-05 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3747 3.9 e33/21 3:965825a6:::rbd_data.115e5ff16b0c.0000000000000038:head v 33'29, mlcod=33'27)", 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9304013710000003, 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003408908, 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887528+0000", 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2240000000000001e-06 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887593+0000", 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4362999999999993e-05 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887606+0000", 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887614+0000", 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2290000000000003e-06 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887680+0000", 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5896999999999996e-05 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887704+0000", 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3516e-05 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890888+0000", 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031837860000000001 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890936+0000", 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8464000000000002e-05 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3763 3.9 e33/21 3:97b67259:::rbd_data.115e5ff16b0c.0000000000000034:head v 33'30, mlcod=33'28)", 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9232741500000001, 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013435750000000001, 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.863 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894656+0000", 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.114e-06 2024-04-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894684+0000", 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8742e-05 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894699+0000", 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5028000000000001e-05 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894708+0000", 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2260000000000005e-06 2024-04-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894771+0000", 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3485000000000006e-05 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894792+0000", 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1393e-05 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.895944+0000", 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001151972 2024-04-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.895998+0000", 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3615000000000001e-05 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3899 3.1f e33/21 3:fb95f7ed:::rbd_data.115e5ff16b0c.000000000000003c:head v 33'29, mlcod=33'27)", 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9196931790000002, 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001093927, 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898237+0000", 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.145e-06 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898269+0000", 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540999999999999e-05 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898285+0000", 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5421e-05 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898292+0000", 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0720000000000002e-06 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898352+0000", 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0182000000000001e-05 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898374+0000", 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2716000000000001e-05 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.899272+0000", 2024-04-24T00:34:09.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089749399999999998 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.899329+0000", 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355999999999998e-05 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:4343 3.4 e33/21 3:276ea59a:::rbd_data.115e5ff16b0c.0000000000000051:head v 33'14, mlcod=30'12)", 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8318189589999996, 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012162010000000001, 2024-04-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986111+0000", 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.336e-06 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986155+0000", 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3664000000000001e-05 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986178+0000", 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3314000000000001e-05 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986188+0000", 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0485e-05 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986254+0000", 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5117999999999996e-05 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986276+0000", 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2586e-05 2024-04-24T00:34:09.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.987285+0000", 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001008695 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.987326+0000", 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1003e-05 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:4451 3.1f e34/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 34'37, mlcod=33'35)", 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:05.541609+0000", 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2763198400000002, 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098417399999999999, 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.541609+0000", 2024-04-24T00:34:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.541609+0000", 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.541609+0000", 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.41e-07 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.541615+0000", 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9259999999999997e-06 2024-04-24T00:34:09.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.541632+0000", 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6755000000000001e-05 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.541641+0000", 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5519999999999993e-06 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.541768+0000", 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126489 2024-04-24T00:34:09.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.541805+0000", 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6819000000000001e-05 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.542550+0000", 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074529999999999996 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.542593+0000", 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2991999999999998e-05 2024-04-24T00:34:09.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:4455 3.1f e34/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 34'38, mlcod=33'36)", 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:05.545939+0000", 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2719897610000004, 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015874050000000001, 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545939+0000", 2024-04-24T00:34:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545939+0000", 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545940+0000", 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.46e-06 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545949+0000", 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:34:09.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545968+0000", 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9840999999999999e-05 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545978+0000", 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7119999999999995e-06 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.546247+0000", 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026843199999999998 2024-04-24T00:34:09.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.546277+0000", 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0614000000000002e-05 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.547463+0000", 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011856659999999999 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.547526+0000", 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3326999999999999e-05 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.1f epoch 36/21 rep_tid 263 entries 36'59 (0'0) error 3:fb1094b8:::rbd_header.115e5ff16b0c:head by client.4446.0:4931 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 35'57)", 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.588455+0000", 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.2294731699999999, 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095937599999999996, 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.588455+0000", 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.588455+0000", 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.588456+0000", 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6400000000000003e-07 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.588468+0000", 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2001000000000001e-05 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.588502+0000", 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3683000000000003e-05 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.588524+0000", 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2248000000000001e-05 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.588690+0000", 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165713 2024-04-24T00:34:09.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.589415+0000", 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072526700000000003 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.173122392, 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:34:09.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:34:09.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:34:09.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5575 3.9 e36/21 3:94eebdaa:::rbd_object_map.115ed6c33904:head v 36'44, mlcod=35'42)", 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.734844+0000", 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0830850510000001, 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001025932, 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.734844+0000", 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.734844+0000", 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.734844+0000", 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4700000000000001e-07 2024-04-24T00:34:09.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.734853+0000", 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.734879+0000", 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6245e-05 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.734894+0000", 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4976e-05 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.735012+0000", 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117689 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.735055+0000", 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2902999999999999e-05 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.735834+0000", 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077940599999999996 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.735869+0000", 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5410000000000001e-05 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5579 3.9 e36/21 3:94eebdaa:::rbd_object_map.115ed6c33904:head v 36'46, mlcod=36'44)", 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.740385+0000", 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.077543433, 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087917699999999998, 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.895 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.740385+0000", 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.740385+0000", 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.740386+0000", 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0200000000000001e-07 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.740391+0000", 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9930000000000003e-06 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.740402+0000", 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0812000000000001e-05 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.740408+0000", 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7689999999999999e-06 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.740469+0000", 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0328000000000002e-05 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.740490+0000", 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1242000000000001e-05 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.741233+0000", 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074281900000000003 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.741264+0000", 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1511999999999999e-05 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5613 3.9 e36/21 3:94eebdaa:::rbd_object_map.115ed6c33904:head v 36'47, mlcod=36'45)", 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.785580+0000", 2024-04-24T00:34:09.899 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.03234819, 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001019697, 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.785580+0000", 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.785580+0000", 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:09.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.785583+0000", 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3999999999999999e-06 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.785624+0000", 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1329999999999999e-05 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.785658+0000", 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4128000000000003e-05 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.785679+0000", 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0574999999999999e-05 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.785763+0000", 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000001e-05 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.785786+0000", 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2435000000000001e-05 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.786521+0000", 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073503999999999996 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.786600+0000", 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9118999999999999e-05 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:09.903 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:09.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:09.908 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:09.904+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:10.009 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:10.004+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:10.026 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:10.026 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:10.026 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:10.026 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:10.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:10.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:10.110 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:10.104+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:10.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:10.195 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:10.196 INFO:teuthology.orchestra.run.smithi064.stdout:625 punch from 0x175ba7f to 0x1762e81, (0x7402 bytes) 2024-04-24T00:34:10.203 INFO:teuthology.orchestra.run.smithi064.stdout:626 writesame 0xbbc6ea thru 0xbc0a86 data_size 0x1(0x439d bytes) 2024-04-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:10.210 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:10.204+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:10.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:10.228 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:10.229 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:10.245 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:10.245 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:10.266 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:10.266 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:10.271 INFO:teuthology.orchestra.run.smithi064.stdout:627 clone 4 order 18 su 2048 sc 3 2024-04-24T00:34:10.287 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:10.287 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:10.307 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:10.307 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:10.311 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:10.308+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:10.327 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:10.328 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:10.330 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:10.330 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:10.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:10.411 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:10.408+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:10.476 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:10.476 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:10.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:10.512 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:10.508+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:10.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:10.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:10.566 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-24T00:34:10.566 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:34:10.566 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:34:10.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-04-24T00:34:10.582 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:34:10.617 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:10.618 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:10.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:10.649 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:10.648+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:10.698 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:10.698 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T00:34:10.698 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:10.713 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:10.708+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:10.750 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:10.750 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:10.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:10.815 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:10.812+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:10.916 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:10.912+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:10.933 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:10.933 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:10.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:11.016 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:11.012+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:11.066 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:11.066 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:11.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:11.117 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:11.116+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:11.218 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:11.212+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:11.240 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:11.240 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:11.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:11.319 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:11.316+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:11.419 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:11.416+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:11.422 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:11.422 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:11.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:11.520 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:11.516+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:11.559 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:11.559 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:11.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:11.621 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:11.620+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:11.722 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:11.716+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:11.746 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:11.746 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:11.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:11.863 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:11.860+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:11.923 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:11.920+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:11.947 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:11.947 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:11.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:12.024 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:12.020+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:12.125 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:12.124+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:12.131 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:12.131 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:12.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:12.226 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:12.220+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:12.258 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:12.258 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:12.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:12.327 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:12.324+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:12.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:12.410 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:12.410 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:12.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:12.427 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:12.424+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:12.497 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:12.513 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:12.514 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:12.528 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:12.524+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:12.531 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:12.531 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:12.546 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:12.546 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:12.546 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:12.550 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:12.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:12.571 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:12.572 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:12.597 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:12.597 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:12.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:34:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:12.623 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:12.629 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:12.624+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:12.646 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:12.646 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:12.704 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:12.704 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:140 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'85)", 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.805253056000002, 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012183782000000001, 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:12.705 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897148+0000", 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897149+0000", 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.147e-06 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897223+0000", 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4116000000000007e-05 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897251+0000", 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7606e-05 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897269+0000", 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7476999999999999e-05 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902553+0000", 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052847359999999999 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.902586+0000", 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2802000000000002e-05 2024-04-24T00:34:12.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909235+0000", 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066493519999999999 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909332+0000", 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6545999999999993e-05 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4204.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'85)", 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.804442946000002, 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011427935, 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897958+0000", 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897960+0000", 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1500000000000002e-06 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.897974+0000", 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3890000000000001e-05 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898000+0000", 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.599e-05 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.898018+0000", 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7903000000000001e-05 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903913+0000", 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058942839999999996 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.903976+0000", 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3847e-05 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909338+0000", 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053611850000000001 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:41.909386+0000", 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8686000000000001e-05 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1964 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 983040~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530433+0000", 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1719682250000005, 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011371305, 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1964 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530433+0000", 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530433+0000", 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530433+0000", 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4799999999999999e-07 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0090000000000004e-06 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530446+0000", 2024-04-24T00:34:12.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1929999999999998e-06 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530450+0000", 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7840000000000003e-06 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541633+0000", 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011182588 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541705+0000", 2024-04-24T00:34:12.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2416999999999995e-05 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8865999999999998e-05 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1981 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530458+0000", 2024-04-24T00:34:12.718 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1719436769999998, 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011545852000000001, 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1981 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530458+0000", 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530458+0000", 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530458+0000", 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0399999999999996e-07 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530461+0000", 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9840000000000001e-06 2024-04-24T00:34:12.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530467+0000", 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1859999999999997e-06 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530480+0000", 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2335e-05 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541833+0000", 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011352948999999999 2024-04-24T00:34:12.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541906+0000", 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3385000000000003e-05 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542003+0000", 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7509000000000004e-05 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1982 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530487+0000", 2024-04-24T00:34:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1719138489999992, 2024-04-24T00:34:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012076564, 2024-04-24T00:34:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1982 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530487+0000", 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530487+0000", 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530488+0000", 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530490+0000", 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3609999999999999e-06 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530497+0000", 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3820000000000001e-06 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530501+0000", 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542356+0000", 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011854131 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542441+0000", 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5604000000000002e-05 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542564+0000", 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012279599999999999 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1983 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1310720~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530508+0000", 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1718931210000001, 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012285136, 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1983 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.728 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530508+0000", 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530508+0000", 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530509+0000", 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530511+0000", 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1280000000000002e-06 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530517+0000", 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0410000000000003e-06 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530521+0000", 2024-04-24T00:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4179999999999999e-06 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542595+0000", 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012073486 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542668+0000", 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3126000000000004e-05 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542793+0000", 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000125573 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1984 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1441792~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530527+0000", 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1718741300000008, 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012414612, 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1984 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530527+0000", 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530527+0000", 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530528+0000", 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899999999999998e-07 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530530+0000", 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0150000000000002e-06 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530536+0000", 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1999999999999999e-06 2024-04-24T00:34:12.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530540+0000", 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4730000000000002e-06 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542825+0000", 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012285007000000001 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542880+0000", 2024-04-24T00:34:12.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4598999999999997e-05 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542942+0000", 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1948999999999999e-05 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1985 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1572864~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.171854776, 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012525493, 2024-04-24T00:34:12.736 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1985 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530547+0000", 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530549+0000", 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.08e-06 2024-04-24T00:34:12.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530555+0000", 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7980000000000002e-06 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530559+0000", 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7380000000000002e-06 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542973+0000", 2024-04-24T00:34:12.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01241387 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543014+0000", 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0392000000000002e-05 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543072+0000", 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8248999999999997e-05 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1986 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1703936~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530567+0000", 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1718347369999993, 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012628393999999999, 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1986 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530567+0000", 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530567+0000", 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530567+0000", 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6199999999999999e-07 2024-04-24T00:34:12.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530569+0000", 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1349999999999999e-06 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530575+0000", 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1489999999999999e-06 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530582+0000", 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543099+0000", 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012517014 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543140+0000", 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0951000000000001e-05 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543195+0000", 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5106999999999999e-05 2024-04-24T00:34:12.744 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1987 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1835008~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1718123760000001, 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012737480000000001, 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.745 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1987 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899999999999998e-07 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530598+0000", 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3909999999999998e-06 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530605+0000", 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4290000000000004e-06 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530610+0000", 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0080000000000002e-06 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543224+0000", 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012613928999999999 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543271+0000", 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7333999999999998e-05 2024-04-24T00:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543326+0000", 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5019999999999998e-05 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:1988 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1966080~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530620+0000", 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1717816229999993, 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012851636, 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1988 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530620+0000", 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530620+0000", 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530620+0000", 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530623+0000", 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2929999999999999e-06 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530629+0000", 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.28e-06 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9699999999999994e-06 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543354+0000", 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012715371 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543394+0000", 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9444999999999999e-05 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543471+0000", 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7700999999999993e-05 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2013 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1114112~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530646+0000", 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1717552740000006, 2024-04-24T00:34:12.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012949534, 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2013 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530646+0000", 2024-04-24T00:34:12.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530646+0000", 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530646+0000", 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8099999999999998e-07 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530649+0000", 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.272e-06 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530655+0000", 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4540000000000004e-06 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530660+0000", 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543500+0000", 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012840014 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543539+0000", 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8652000000000002e-05 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543596+0000", 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6777000000000002e-05 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2014 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1245184~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530667+0000", 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1717343800000002, 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013051086, 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2014 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530667+0000", 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530667+0000", 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530667+0000", 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2899999999999999e-07 2024-04-24T00:34:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530669+0000", 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530676+0000", 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1650000000000003e-06 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530680+0000", 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4719999999999997e-06 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543623+0000", 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012943378 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543662+0000", 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8058000000000001e-05 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543718+0000", 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6431000000000002e-05 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2015 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1376256~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1717148579999996, 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013207162, 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2015 2024-04-24T00:34:12.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530687+0000", 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8799999999999998e-07 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.001e-06 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530695+0000", 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2589999999999997e-06 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530700+0000", 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5310000000000003e-06 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543744+0000", 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013044616 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543795+0000", 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1084000000000001e-05 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543894+0000", 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8282999999999994e-05 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2016 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1507328~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530707+0000", 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1716938809999995, 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013377231999999999, 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.766 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2016 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530707+0000", 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530707+0000", 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530708+0000", 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530714+0000", 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0589999999999999e-06 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530720+0000", 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2550000000000003e-06 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530725+0000", 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6190000000000002e-06 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543923+0000", 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0131988 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543995+0000", 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1420000000000004e-05 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544085+0000", 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9711999999999998e-05 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2017 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1638400~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530731+0000", 2024-04-24T00:34:12.770 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1716700509999995, 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013532136, 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2017 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530731+0000", 2024-04-24T00:34:12.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530731+0000", 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530732+0000", 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1300000000000001e-07 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530739+0000", 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6070000000000002e-06 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530746+0000", 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9009999999999997e-06 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530751+0000", 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0810000000000001e-06 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544115+0000", 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013363284 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544167+0000", 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2419000000000001e-05 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544263+0000", 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6430999999999999e-05 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2018 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1769472~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1716430869999996, 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013700711000000001, 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2018 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530759+0000", 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7099999999999997e-07 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530761+0000", 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1550000000000001e-06 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530767+0000", 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1639999999999998e-06 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530772+0000", 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.848e-06 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544294+0000", 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013521948000000001 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544368+0000", 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4207000000000003e-05 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544459+0000", 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1018000000000002e-05 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2019 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1716229170000005, 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014241181, 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2019 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530779+0000", 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530780+0000", 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.686e-06 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.507e-06 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530790+0000", 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5240000000000002e-06 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544487+0000", 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013696612 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544570+0000", 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2704e-05 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545020+0000", 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044978100000000001 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.782 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2020 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1716043319999994, 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014215145, 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2020 2024-04-24T00:34:12.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9499999999999998e-07 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530799+0000", 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.013e-06 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4289999999999998e-06 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530810+0000", 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4719999999999997e-06 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544812+0000", 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01400174 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544855+0000", 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3217999999999999e-05 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545012+0000", 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000156878 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:3707 3.1 3:860763fd:::rbd_data.115e5ff16b0c.000000000000002d:head [zero 0~141397,write 141397~52132 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.869574+0000", 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.8328277179999999, 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012924814, 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:12.787 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3707 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869574+0000", 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:12.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869574+0000", 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869575+0000", 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7499999999999998e-07 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869625+0000", 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0260000000000003e-05 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:12.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869638+0000", 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3706999999999999e-05 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869658+0000", 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869731+0000", 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2564999999999994e-05 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.869886+0000", 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582399999999999 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.870086+0000", 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000199507 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.874167+0000", 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040809339999999996 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:12.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.882342+0000", 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081753469999999995 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.882403+0000", 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0705999999999998e-05 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.882498+0000", 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5451999999999993e-05 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:12.793 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:12.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:12.795 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-04-24T00:34:12.795 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:12.788+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:12.895 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:12.892+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:12.962 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:12.963 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:12.963 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:12.977 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:13.021 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:13.020+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:13.097 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:13.096+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:13.143 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:13.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:13.171 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:629 trunc from 0xe8e14b1 to 0xb156a1f 2024-04-24T00:34:13.197 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:13.196+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:13.203 INFO:teuthology.orchestra.run.smithi064.stdout:630 writesame 0x1d06f74 thru 0x1d133ee data_size 0x1(0xc47b bytes) 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2025 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1638400~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7601938970000006, 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077913280000000001, 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2025 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530291+0000", 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3200000000000005e-07 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530294+0000", 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530303+0000", 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8820000000000007e-06 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530310+0000", 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0199999999999997e-06 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.537775+0000", 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074650569999999998 2024-04-24T00:34:13.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.537853+0000", 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7961999999999997e-05 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538082+0000", 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0002288 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2026 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1769472~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7601647370000002, 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081592609999999993, 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2026 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.298 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530320+0000", 2024-04-24T00:34:13.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6700000000000003e-07 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530323+0000", 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7420000000000002e-06 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530332+0000", 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6059999999999992e-06 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530340+0000", 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000001e-06 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538180+0000", 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078402369999999999 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538308+0000", 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012834000000000001 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538479+0000", 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170507 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2027 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7601347720000007, 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.009211136, 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.302 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2027 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530350+0000", 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7199999999999999e-07 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530353+0000", 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3390000000000001e-06 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530362+0000", 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0489999999999993e-06 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530370+0000", 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5739999999999998e-06 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539401+0000", 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090311609999999994 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539466+0000", 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4880000000000002e-05 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539561+0000", 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4561000000000005e-05 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2028 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7601042489999994, 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095022319999999993, 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2028 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7899999999999998e-07 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530384+0000", 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0280000000000001e-06 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530392+0000", 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7830000000000004e-06 2024-04-24T00:34:13.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530400+0000", 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2080000000000003e-06 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539616+0000", 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092167310000000006 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539722+0000", 2024-04-24T00:34:13.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010527900000000001 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539882+0000", 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000160624 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2045 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7600741860000007, 2024-04-24T00:34:13.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097538699999999996, 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2045 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530411+0000", 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530414+0000", 2024-04-24T00:34:13.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2430000000000001e-06 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530423+0000", 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3030000000000005e-06 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530430+0000", 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.917e-06 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539923+0000", 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094929349999999992 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540027+0000", 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010355 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540164+0000", 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013730799999999999 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2046 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7600439800000007, 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010030647, 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2046 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.315 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530441+0000", 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3200000000000005e-07 2024-04-24T00:34:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530444+0000", 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8109999999999999e-06 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530453+0000", 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3409999999999992e-06 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530460+0000", 2024-04-24T00:34:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3719999999999999e-06 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540209+0000", 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097491539999999995 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540326+0000", 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001166 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540471+0000", 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014483700000000001 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2047 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7600133109999998, 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010295987, 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2047 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530472+0000", 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2799999999999995e-07 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530475+0000", 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0939999999999999e-06 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530485+0000", 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7969999999999995e-06 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530491+0000", 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8709999999999999e-06 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540514+0000", 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010022499000000001 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540626+0000", 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111981 2024-04-24T00:34:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540767+0000", 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014091700000000001 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2048 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7599824999999996, 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010591068, 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2048 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2699999999999999e-07 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530520+0000", 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8165999999999999e-05 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530533+0000", 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2065999999999999e-05 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540812+0000", 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010266239 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540925+0000", 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112676 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541093+0000", 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016786500000000001 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2049 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.759926257, 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010815047, 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2049 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530562+0000", 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4369999999999999e-06 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530572+0000", 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530579+0000", 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541138+0000", 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010558209000000001 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541239+0000", 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101483 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541373+0000", 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133769 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2050 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7598949079999997, 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011052364, 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2050 2024-04-24T00:34:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530590+0000", 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6599999999999996e-07 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530593+0000", 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.937e-06 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530602+0000", 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4360000000000008e-06 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530609+0000", 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0090000000000001e-06 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541411+0000", 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01080191 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541511+0000", 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9839000000000004e-05 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541642+0000", 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130667 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2051 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7598655480000005, 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011318002000000001, 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2051 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2900000000000004e-07 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530622+0000", 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9490000000000001e-06 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530632+0000", 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999994e-06 2024-04-24T00:34:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.131e-06 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541683+0000", 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011044629 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120383 2024-04-24T00:34:13.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541937+0000", 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132941 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2052 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7598363480000003, 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011583872, 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2052 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.44e-07 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530658+0000", 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9850000000000001e-06 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530668+0000", 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530675+0000", 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6240000000000002e-06 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541978+0000", 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011302921 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542100+0000", 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121593 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542232+0000", 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013176600000000001 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2077 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.344 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7597989480000003, 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011842221, 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2077 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:13.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6000000000000003e-07 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.095e-06 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530699+0000", 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530706+0000", 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2429999999999999e-06 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542275+0000", 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011568746 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542386+0000", 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011176200000000001 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542527+0000", 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014114499999999999 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2078 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7597686219999993, 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012081010999999999, 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2078 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.349 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:13.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2200000000000004e-07 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530719+0000", 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9349999999999999e-06 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530728+0000", 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0860000000000008e-06 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530735+0000", 2024-04-24T00:34:13.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2169999999999997e-06 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542568+0000", 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011832540000000001 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542667+0000", 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9000999999999996e-05 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542797+0000", 2024-04-24T00:34:13.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012971 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2079 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7597393090000004, 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012334128999999999, 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2079 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530748+0000", 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9040000000000001e-06 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2159999999999995e-06 2024-04-24T00:34:13.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530764+0000", 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9890000000000003e-06 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542839+0000", 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012074354000000001 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542945+0000", 2024-04-24T00:34:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106443 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543079+0000", 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013369599999999999 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2080 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7597105089999996, 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012588825999999999, 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.357 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2080 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530777+0000", 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.022e-06 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:13.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9879999999999993e-06 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530793+0000", 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1409999999999999e-06 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543122+0000", 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012328274 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543230+0000", 2024-04-24T00:34:13.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108046 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543362+0000", 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013286799999999999 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2081 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7596816240000006, 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012843515, 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2081 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1500000000000005e-07 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.819e-06 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530815+0000", 2024-04-24T00:34:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3540000000000006e-06 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530835+0000", 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9983000000000002e-05 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543405+0000", 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012569739 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543513+0000", 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010821500000000001 2024-04-24T00:34:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543646+0000", 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289000000000001 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2082 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7596384090000008, 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013081859, 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2082 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530849+0000", 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0750000000000002e-06 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530859+0000", 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7079999999999992e-06 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530866+0000", 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4309999999999997e-06 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543689+0000", 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012822750000000001 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543796+0000", 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106526 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543928+0000", 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131846 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2083 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7596081360000007, 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013453964000000001, 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2083 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530877+0000", 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530887+0000", 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186e-05 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530896+0000", 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5480000000000007e-06 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530904+0000", 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4599999999999997e-06 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543970+0000", 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013066067000000001 2024-04-24T00:34:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544178+0000", 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020785000000000001 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544330+0000", 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015230699999999999 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2084 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7595703119999992, 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907047000000001, 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2084 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3099999999999998e-07 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530922+0000", 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.498e-06 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530932+0000", 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530940+0000", 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4769999999999997e-06 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544421+0000", 2024-04-24T00:34:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013481739 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544673+0000", 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025143600000000001 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544821+0000", 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014819 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.378 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:13.378 INFO:teuthology.orchestra.run.smithi064.stdout:631 write 0xaf69f51 thru 0xaf75036 (0xb0e6 bytes) 2024-04-24T00:34:13.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:632 writesame 0xb99989b thru 0xb9a921a data_size 0x40(0xf980 bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:633 punch from 0xdddf94 to 0xde8058, (0xa0c4 bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:634 read 0x5c688fa thru 0x5c69ab3 (0x11ba bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:635 writesame 0x5efeac4 thru 0x5eff177 data_size 0x21(0x6b4 bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:638 read 0xba82ce thru 0xbaaad5 (0x2808 bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:639 writesame 0x2b5f43 thru 0x2b8c38 data_size 0x5(0x2cf6 bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:640 write 0x961daff thru 0x96277b1 (0x9cb3 bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:641 read 0x2b4a803 thru 0x2b4dac0 (0x32be bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:642 read 0x2ff8929 thru 0x2ffe4ce (0x5ba6 bytes) 2024-04-24T00:34:13.381 INFO:teuthology.orchestra.run.smithi064.stdout:643 punch from 0x5883521 to 0x5885299, (0x1d78 bytes) 2024-04-24T00:34:13.382 INFO:teuthology.orchestra.run.smithi064.stdout:644 write 0x6d60718 thru 0x6d6d0d0 (0xc9b9 bytes) 2024-04-24T00:34:13.382 INFO:teuthology.orchestra.run.smithi064.stdout:645 read 0x6556e94 thru 0x6557814 (0x981 bytes) 2024-04-24T00:34:13.382 INFO:teuthology.orchestra.run.smithi064.stdout:647 read 0x39a9ac8 thru 0x39b91cc (0xf705 bytes) 2024-04-24T00:34:13.382 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:13.382 INFO:teuthology.orchestra.run.smithi064.stdout:649 read 0x9da9d48 thru 0x9db92e5 (0xf59e bytes) 2024-04-24T00:34:13.382 INFO:teuthology.orchestra.run.smithi064.stdout:651 trunc from 0xb9a921b to 0x88109e9 2024-04-24T00:34:13.382 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:13.376+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:13.389 INFO:teuthology.orchestra.run.smithi064.stdout:652 punch from 0x3519698 to 0x351abc0, (0x1528 bytes) 2024-04-24T00:34:13.389 INFO:teuthology.orchestra.run.smithi064.stdout:653 read 0x7f690 thru 0x80852 (0x11c3 bytes) 2024-04-24T00:34:13.391 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:13.407 INFO:teuthology.orchestra.run.smithi064.stdout:655 writesame 0xc04ecbd thru 0xc051f3a data_size 0x17(0x327e bytes) 2024-04-24T00:34:13.408 INFO:teuthology.orchestra.run.smithi064.stdout:656 write 0x854c1fe thru 0x8557b94 (0xb997 bytes) 2024-04-24T00:34:13.421 INFO:teuthology.orchestra.run.smithi064.stdout:657 writesame 0xa2a1a65 thru 0xa2a4ec1 data_size 0x7(0x345d bytes) 2024-04-24T00:34:13.432 INFO:teuthology.orchestra.run.smithi064.stdout:658 read 0x84c1572 thru 0x84c4a47 (0x34d6 bytes) 2024-04-24T00:34:13.435 INFO:teuthology.orchestra.run.smithi064.stdout:660 trunc from 0xc051f3b to 0xcca9ded 2024-04-24T00:34:13.444 INFO:teuthology.orchestra.run.smithi064.stdout:663 writesame 0x5df0051 thru 0x5df8027 data_size 0x1(0x7fd7 bytes) 2024-04-24T00:34:13.479 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:13.476+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:13.496 INFO:teuthology.orchestra.run.smithi064.stdout:664 read 0xc3c58b5 thru 0xc3c8f9d (0x36e9 bytes) 2024-04-24T00:34:13.496 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:13.496 INFO:teuthology.orchestra.run.smithi064.stdout:667 write 0x7f6556d thru 0x7f6730e (0x1da2 bytes) 2024-04-24T00:34:13.509 INFO:teuthology.orchestra.run.smithi064.stdout:668 writesame 0x8e011c4 thru 0x8e04ab2 data_size 0x1(0x38ef bytes) 2024-04-24T00:34:13.519 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:13.519 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:13.519 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:13.534 INFO:teuthology.orchestra.run.smithi064.stdout:669 trunc from 0xcca9ded to 0x938c105 2024-04-24T00:34:13.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:13.558 INFO:teuthology.orchestra.run.smithi064.stdout:670 trunc from 0x938c105 to 0xc08eb7d 2024-04-24T00:34:13.577 INFO:teuthology.orchestra.run.smithi064.stdout:672 write 0xc335b3d thru 0xc33b05b (0x551f bytes) 2024-04-24T00:34:13.580 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:13.576+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:13.599 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:34:13.608 INFO:teuthology.orchestra.run.smithi064.stdout:673 trunc from 0xc33b05c to 0x50b5394 2024-04-24T00:34:13.644 INFO:teuthology.orchestra.run.smithi064.stdout:675 punch from 0x2016df to 0x201b62, (0x483 bytes) 2024-04-24T00:34:13.644 INFO:teuthology.orchestra.run.smithi064.stdout:676 read 0x38afa54 thru 0x38b4a10 (0x4fbd bytes) 2024-04-24T00:34:13.644 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:13.657 INFO:teuthology.orchestra.run.smithi064.stdout:679 writesame 0x906280a thru 0x906ef81 data_size 0x34(0xc778 bytes) 2024-04-24T00:34:13.657 INFO:teuthology.orchestra.run.smithi064.stdout:680 punch from 0x65717bc to 0x6575cef, (0x4533 bytes) 2024-04-24T00:34:13.657 INFO:teuthology.orchestra.run.smithi064.stdout:681 writesame 0x387951a thru 0x387a007 data_size 0x1(0xaee bytes) 2024-04-24T00:34:13.669 INFO:teuthology.orchestra.run.smithi064.stdout:685 writesame 0xb73438 thru 0xb7b213 data_size 0x5a(0x7ddc bytes) 2024-04-24T00:34:13.674 INFO:teuthology.orchestra.run.smithi064.stdout:687 write 0x63b6390 thru 0x63bf68d (0x92fe bytes) 2024-04-24T00:34:13.681 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:13.676+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:13.687 INFO:teuthology.orchestra.run.smithi064.stdout:688 write 0x7fa56fd thru 0x7faf2f5 (0x9bf9 bytes) 2024-04-24T00:34:13.689 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:13.689 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:13.690 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:13.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:13.712 INFO:teuthology.orchestra.run.smithi064.stdout:689 write 0xb06f3f2 thru 0xb079c63 (0xa872 bytes) 2024-04-24T00:34:13.722 INFO:teuthology.orchestra.run.smithi064.stdout:691 punch from 0x3d83f7e to 0x3d8fd0a, (0xbd8c bytes) 2024-04-24T00:34:13.722 INFO:teuthology.orchestra.run.smithi064.stdout:693 write 0x95c98d4 thru 0x95cb2b4 (0x19e1 bytes) 2024-04-24T00:34:13.730 INFO:teuthology.orchestra.run.smithi064.stdout:695 writesame 0x50a4566 thru 0x50a94bd data_size 0x4(0x4f58 bytes) 2024-04-24T00:34:13.742 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:13.742 INFO:teuthology.orchestra.run.smithi064.stdout:703 read 0x7d7fc9f thru 0x7d82437 (0x2799 bytes) 2024-04-24T00:34:13.750 INFO:teuthology.orchestra.run.smithi064.stdout:704 write 0xb4ebe47 thru 0xb4f55f5 (0x97af bytes) 2024-04-24T00:34:13.774 INFO:teuthology.orchestra.run.smithi064.stdout:705 trunc from 0xb4f55f6 to 0xb10c47 2024-04-24T00:34:13.782 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:13.780+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:13.831 INFO:teuthology.orchestra.run.smithi064.stdout:707 trunc from 0xb10c47 to 0xdbbfae9 2024-04-24T00:34:13.840 INFO:teuthology.orchestra.run.smithi064.stdout:709 read 0x13518d7 thru 0x135566e (0x3d98 bytes) 2024-04-24T00:34:13.840 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:13.840 INFO:teuthology.orchestra.run.smithi064.stdout:713 write 0x50c5168 thru 0x50c717b (0x2014 bytes) 2024-04-24T00:34:13.849 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:11 3.1f e24/21 3:fa654f18:::rbd_id.image_client.0:head v 24'1, mlcod=0'0)", 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.808130115000001, 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001106349, 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037945+0000", 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.06e-07 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037959+0000", 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4007e-05 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037974+0000", 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4586999999999999e-05 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.037985+0000", 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.137e-05 2024-04-24T00:34:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.038111+0000", 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012570699999999999 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.038143+0000", 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2218999999999998e-05 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039003+0000", 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085975500000000002 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:33:55.039051+0000", 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8097999999999999e-05 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2348 3.1f e33/21 3:fac32ee7:::rbd_data.115e5ff16b0c.0000000000000009:head v 33'26, mlcod=32'24)", 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2732876209999997, 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011498890000000001, 2024-04-24T00:34:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572787+0000", 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572788+0000", 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572816+0000", 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8061000000000001e-05 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572827+0000", 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1423999999999999e-05 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572835+0000", 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.858e-06 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572898+0000", 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3139000000000007e-05 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.572929+0000", 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0355999999999999e-05 2024-04-24T00:34:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.573887+0000", 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095867599999999999 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.573937+0000", 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0059999999999998e-05 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2984 3.5 e33/21 3:a2b34f77:::rbd_data.115e5ff16b0c.000000000000001c:head v 33'37, mlcod=30'35)", 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2325687849999998, 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437279999999999, 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613507+0000", 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.63e-07 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613521+0000", 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4545999999999999e-05 2024-04-24T00:34:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613540+0000", 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8963000000000001e-05 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613550+0000", 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0298e-05 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613622+0000", 2024-04-24T00:34:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1476999999999993e-05 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613645+0000", 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2940999999999999e-05 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.615112+0000", 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014666729999999999 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.615150+0000", 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8367e-05 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2979 3.9 e33/21 3:9717bf35:::rbd_data.115e5ff16b0c.000000000000001b:head v 33'28, mlcod=30'26)", 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0238734439999995, 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053184019999999999, 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822203+0000", 2024-04-24T00:34:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3909999999999999e-06 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822482+0000", 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027925399999999999 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822509+0000", 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.707e-05 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822521+0000", 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1779e-05 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822626+0000", 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105165 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822653+0000", 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6713e-05 2024-04-24T00:34:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827477+0000", 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048240949999999996 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827520+0000", 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2935000000000001e-05 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.1f epoch 33/21 rep_tid 217 entries 33'27 (0'0) error 3:fb1094b8:::rbd_header.115e5ff16b0c:head by client.4446.0:3392 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 32'25)", 2024-04-24T00:34:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0072839519999999, 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001188705, 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838791+0000", 2024-04-24T00:34:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838792+0000", 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838804+0000", 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1861e-05 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838830+0000", 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5428000000000001e-05 2024-04-24T00:34:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838845+0000", 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5729000000000001e-05 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.838928+0000", 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2584999999999996e-05 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.839980+0000", 2024-04-24T00:34:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001051754 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3748 3.1f e33/21 3:feb5d058:::rbd_data.115e5ff16b0c.0000000000000030:head v 33'28, mlcod=33'26)", 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9670852409999995, 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028352379999999999, 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878991+0000", 2024-04-24T00:34:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879014+0000", 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2677e-05 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879033+0000", 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9361999999999999e-05 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879043+0000", 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2299999999999997e-06 2024-04-24T00:34:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879110+0000", 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7385999999999995e-05 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879132+0000", 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2405e-05 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881782+0000", 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026490979999999999 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881825+0000", 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3460000000000001e-05 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3747 3.9 e33/21 3:965825a6:::rbd_data.115e5ff16b0c.0000000000000038:head v 33'29, mlcod=33'27)", 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9585477900000008, 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003408908, 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887528+0000", 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2240000000000001e-06 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887593+0000", 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4362999999999993e-05 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887606+0000", 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887614+0000", 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2290000000000003e-06 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887680+0000", 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5896999999999996e-05 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887704+0000", 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3516e-05 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890888+0000", 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031837860000000001 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890936+0000", 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8464000000000002e-05 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3763 3.9 e33/21 3:97b67259:::rbd_data.115e5ff16b0c.0000000000000034:head v 33'30, mlcod=33'28)", 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9514205689999997, 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013435750000000001, 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894654+0000", 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894656+0000", 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.114e-06 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894684+0000", 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8742e-05 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894699+0000", 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5028000000000001e-05 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894708+0000", 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2260000000000005e-06 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894771+0000", 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3485000000000006e-05 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.894792+0000", 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1393e-05 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.895944+0000", 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001151972 2024-04-24T00:34:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.895998+0000", 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3615000000000001e-05 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3899 3.1f e33/21 3:fb95f7ed:::rbd_data.115e5ff16b0c.000000000000003c:head v 33'29, mlcod=33'27)", 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9478395979999998, 2024-04-24T00:34:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001093927, 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898235+0000", 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898237+0000", 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.145e-06 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898269+0000", 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540999999999999e-05 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898285+0000", 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5421e-05 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898292+0000", 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0720000000000002e-06 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898352+0000", 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0182000000000001e-05 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.898374+0000", 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2716000000000001e-05 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.899272+0000", 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089749399999999998 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.899329+0000", 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355999999999998e-05 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:4343 3.4 e33/21 3:276ea59a:::rbd_data.115e5ff16b0c.0000000000000051:head v 33'14, mlcod=30'12)", 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8599653780000001, 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012162010000000001, 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986110+0000", 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986111+0000", 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.336e-06 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986155+0000", 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3664000000000001e-05 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986178+0000", 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3314000000000001e-05 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986188+0000", 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0485e-05 2024-04-24T00:34:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986254+0000", 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5117999999999996e-05 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.986276+0000", 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2586e-05 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.987285+0000", 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001008695 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.987326+0000", 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1003e-05 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:4455 3.1f e34/21 3:fb1094b8:::rbd_header.115e5ff16b0c:head v 34'38, mlcod=33'36)", 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:05.545939+0000", 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.3001361799999991, 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015874050000000001, 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545939+0000", 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545939+0000", 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545940+0000", 2024-04-24T00:34:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.46e-06 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545949+0000", 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545968+0000", 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9840999999999999e-05 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.545978+0000", 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7119999999999995e-06 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.546247+0000", 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026843199999999998 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.546277+0000", 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0614000000000002e-05 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.547463+0000", 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011856659999999999 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:05.547526+0000", 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3326999999999999e-05 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2012688110000003, 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:34:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:34:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10074 3.5 e40/21 3:a07ee5e3:::rbd_header.115e380d621c:head v 40'61, mlcod=39'59)", 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:12.599201+0000", 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.2468744039999999, 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011057899999999999, 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.599201+0000", 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.599201+0000", 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.599201+0000", 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9800000000000003e-07 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.599209+0000", 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5070000000000003e-06 2024-04-24T00:34:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.599226+0000", 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7031999999999999e-05 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.599234+0000", 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.599301+0000", 2024-04-24T00:34:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6657999999999998e-05 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.599328+0000", 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7494000000000001e-05 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.600270+0000", 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094144900000000002 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:12.600306+0000", 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6850999999999997e-05 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10260 3.9 e40/21 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head v 40'55, mlcod=38'53)", 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.62071885699999996, 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027588550000000002, 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225357+0000", 2024-04-24T00:34:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3319999999999999e-06 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226434+0000", 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010764730000000001 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226460+0000", 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6478e-05 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226476+0000", 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5769000000000001e-05 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226662+0000", 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186202 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226706+0000", 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3760000000000001e-05 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.228025+0000", 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013191699999999999 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.228115+0000", 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9671000000000003e-05 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10336 3.9 e40/21 3:912611d7:::rbd_data.115e380d621c.0000000000000213:head v 40'56, mlcod=39'54)", 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.43036703399999998, 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021373809999999998, 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415710+0000", 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.973e-06 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415797+0000", 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7194999999999994e-05 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415825+0000", 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7501999999999999e-05 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415838+0000", 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3232e-05 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415929+0000", 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1360000000000006e-05 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415962+0000", 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2570999999999997e-05 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.417692+0000", 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001729801 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.417845+0000", 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153747 2024-04-24T00:34:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10361 3.1f e40/21 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head v 40'65, mlcod=40'63)", 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.38985498000000002, 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020517270000000002, 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456223+0000", 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6970000000000002e-06 2024-04-24T00:34:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456269+0000", 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.617e-05 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456302+0000", 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3028000000000003e-05 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456323+0000", 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-04-24T00:34:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456451+0000", 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012818899999999999 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456512+0000", 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0680000000000002e-05 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458154+0000", 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016422330000000001 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458272+0000", 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117477 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10425 3.4 e41/21 3:2657577c:::rbd_data.115e380d621c.0000000000000210:head v 41'28, mlcod=39'26)", 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.22779813700000001, 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041008809999999998, 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618285+0000", 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5580000000000003e-06 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618309+0000", 2024-04-24T00:34:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-05 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618321+0000", 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.164e-05 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619328+0000", 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010071259999999999 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619360+0000", 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1581000000000001e-05 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622325+0000", 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029648610000000001 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622378+0000", 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3235000000000001e-05 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10415 3.1f e41/21 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head v 41'66, mlcod=40'64)", 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.22655942100000001, 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029607800000000001, 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.53e-07 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619520+0000", 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3710000000000002e-06 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619530+0000", 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4609999999999997e-06 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619536+0000", 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2879999999999998e-06 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619618+0000", 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2091999999999999e-05 2024-04-24T00:34:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619640+0000", 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2042e-05 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622396+0000", 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027557300000000001 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622476+0000", 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0543000000000005e-05 2024-04-24T00:34:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10531 3.9 e41/21 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head v 41'60, mlcod=41'58)", 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.066414947000000002, 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030187019999999998, 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779662+0000", 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6819999999999999e-06 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779672+0000", 2024-04-24T00:34:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0482999999999999e-05 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779700+0000", 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8218999999999999e-05 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779713+0000", 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2123000000000001e-05 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779827+0000", 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114355 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779866+0000", 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9449000000000001e-05 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.782618+0000", 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027520980000000001 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.782679+0000", 2024-04-24T00:34:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0293e-05 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10541 3.1f e41/21 3:f85c3e31:::rbd_data.115e380d621c.0000000000000141:head v 41'69, mlcod=41'67)", 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.060597438000000003, 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025351169999999999, 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785480+0000", 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2879999999999999e-06 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785493+0000", 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3036e-05 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785529+0000", 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6043e-05 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785546+0000", 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6897000000000001e-05 2024-04-24T00:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785630+0000", 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4124999999999998e-05 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785657+0000", 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.724e-05 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.787959+0000", 2024-04-24T00:34:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023013920000000002 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.788013+0000", 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4095999999999998e-05 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:13.936 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:13.937 INFO:teuthology.orchestra.run.smithi064.stdout:715 writesame 0x4719c95 thru 0x4722f09 data_size 0x1(0x9275 bytes) 2024-04-24T00:34:13.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:13.939 INFO:teuthology.orchestra.run.smithi064.stdout:716 write 0xa32e7db thru 0xa33d829 (0xf04f bytes) 2024-04-24T00:34:13.939 INFO:teuthology.orchestra.run.smithi064.stdout:717 read 0xadeb0a9 thru 0xadf2897 (0x77ef bytes) 2024-04-24T00:34:13.939 INFO:teuthology.orchestra.run.smithi064.stdout:718 write 0x706d735 thru 0x7070650 (0x2f1c bytes) 2024-04-24T00:34:13.939 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:13.936+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:13.947 INFO:teuthology.orchestra.run.smithi064.stdout:719 write 0xbd80b1c thru 0xbd832be (0x27a3 bytes) 2024-04-24T00:34:13.971 INFO:teuthology.orchestra.run.smithi064.stdout:720 trunc from 0xdbbfae9 to 0x88ab28f 2024-04-24T00:34:13.993 INFO:teuthology.orchestra.run.smithi064.stdout:721 read 0x37d6968 thru 0x37e4e53 (0xe4ec bytes) 2024-04-24T00:34:13.993 INFO:teuthology.orchestra.run.smithi064.stdout:722 writesame 0x233d88a thru 0x233f2f7 data_size 0x11(0x1a6e bytes) 2024-04-24T00:34:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:723 write 0xe1f68d0 thru 0xe1fc6e9 (0x5e1a bytes) 2024-04-24T00:34:14.038 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:14.036+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:14.055 INFO:teuthology.orchestra.run.smithi064.stdout:724 write 0xe3ccfb9 thru 0xe3cd5f8 (0x640 bytes) 2024-04-24T00:34:14.064 INFO:teuthology.orchestra.run.smithi064.stdout:725 write 0x3f4d07b thru 0x3f54ca5 (0x7c2b bytes) 2024-04-24T00:34:14.079 INFO:teuthology.orchestra.run.smithi064.stdout:726 punch from 0x63ee023 to 0x63f253a, (0x4517 bytes) 2024-04-24T00:34:14.079 INFO:teuthology.orchestra.run.smithi064.stdout:728 read 0xd842369 thru 0xd84c22e (0x9ec6 bytes) 2024-04-24T00:34:14.079 INFO:teuthology.orchestra.run.smithi064.stdout:730 read 0xdc7d36c thru 0xdc80561 (0x31f6 bytes) 2024-04-24T00:34:14.079 INFO:teuthology.orchestra.run.smithi064.stdout:732 punch from 0x8573529 to 0x857e133, (0xac0a bytes) 2024-04-24T00:34:14.079 INFO:teuthology.orchestra.run.smithi064.stdout:734 write 0xad4329f thru 0xad465c4 (0x3326 bytes) 2024-04-24T00:34:14.090 INFO:teuthology.orchestra.run.smithi064.stdout:736 read 0x193eb50 thru 0x194aef2 (0xc3a3 bytes) 2024-04-24T00:34:14.091 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:14.091 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:14.091 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:14.106 INFO:teuthology.orchestra.run.smithi064.stdout:738 trunc from 0xe3cd5f9 to 0x783d2ec 2024-04-24T00:34:14.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:14.129 INFO:teuthology.orchestra.run.smithi064.stdout:739 read 0x2c23872 thru 0x2c2e9aa (0xb139 bytes) 2024-04-24T00:34:14.140 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:14.136+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:14.154 INFO:teuthology.orchestra.run.smithi064.stdout:740 write 0xe647177 thru 0xe64c7b0 (0x563a bytes) 2024-04-24T00:34:14.166 INFO:teuthology.orchestra.run.smithi064.stdout:741 read 0x30ab8bc thru 0x30ad2c9 (0x1a0e bytes) 2024-04-24T00:34:14.167 INFO:teuthology.orchestra.run.smithi064.stdout:743 read 0x46f9b81 thru 0x4706a47 (0xcec7 bytes) 2024-04-24T00:34:14.168 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:14.168 INFO:teuthology.orchestra.run.smithi064.stdout:746 punch from 0x679c1c4 to 0x67a585d, (0x9699 bytes) 2024-04-24T00:34:14.168 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:14.168 INFO:teuthology.orchestra.run.smithi064.stdout:749 writesame 0x5f96988 thru 0x5f9cb49 data_size 0x61(0x61c2 bytes) 2024-04-24T00:34:14.179 INFO:teuthology.orchestra.run.smithi064.stdout:750 writesame 0x85f07ac thru 0x85fb2d2 data_size 0xf(0xab27 bytes) 2024-04-24T00:34:14.194 INFO:teuthology.orchestra.run.smithi064.stdout:751 punch from 0x37a2ad0 to 0x37a5172, (0x26a2 bytes) 2024-04-24T00:34:14.194 INFO:teuthology.orchestra.run.smithi064.stdout:753 read 0x9b5fec0 thru 0x9b6d66b (0xd7ac bytes) 2024-04-24T00:34:14.194 INFO:teuthology.orchestra.run.smithi064.stdout:755 punch from 0x6ff0db5 to 0x6ff0fcc, (0x217 bytes) 2024-04-24T00:34:14.194 INFO:teuthology.orchestra.run.smithi064.stdout:756 write 0x796b32d thru 0x796e59a (0x326e bytes) 2024-04-24T00:34:14.205 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:14.205 INFO:teuthology.orchestra.run.smithi064.stdout:758 writesame 0x8773e6a thru 0x877e4e4 data_size 0x17(0xa67b bytes) 2024-04-24T00:34:14.216 INFO:teuthology.orchestra.run.smithi064.stdout:760 writesame 0x66a66ce thru 0x66ab1c2 data_size 0x1(0x4af5 bytes) 2024-04-24T00:34:14.238 INFO:teuthology.orchestra.run.smithi064.stdout:761 trunc from 0xe64c7b1 to 0xb9ef230 2024-04-24T00:34:14.240 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:14.236+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:14.258 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:14.258 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:14.258 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:14.258 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:14.258 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:14.259 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:14.259 INFO:teuthology.orchestra.run.smithi064.stdout:763 write 0x10f8af2 thru 0x1100f0a (0x8419 bytes) 2024-04-24T00:34:14.270 INFO:teuthology.orchestra.run.smithi064.stdout:764 writesame 0x8fd4afc thru 0x8fd6222 data_size 0x1(0x1727 bytes) 2024-04-24T00:34:14.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:14.284 INFO:teuthology.orchestra.run.smithi064.stdout:765 punch from 0x590ce65 to 0x5914a76, (0x7c11 bytes) 2024-04-24T00:34:14.284 INFO:teuthology.orchestra.run.smithi064.stdout:766 read 0xb1465b9 thru 0xb150bf3 (0xa63b bytes) 2024-04-24T00:34:14.284 INFO:teuthology.orchestra.run.smithi064.stdout:767 write 0x4bbeffe thru 0x4bcc489 (0xd48c bytes) 2024-04-24T00:34:14.305 INFO:teuthology.orchestra.run.smithi064.stdout:768 punch from 0x9181b3d to 0x918d3b0, (0xb873 bytes) 2024-04-24T00:34:14.305 INFO:teuthology.orchestra.run.smithi064.stdout:769 writesame 0x151de76 thru 0x152aa0f data_size 0x33(0xcb9a bytes) 2024-04-24T00:34:14.315 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:14.315 INFO:teuthology.orchestra.run.smithi064.stdout:771 punch from 0x71281a to 0x714f5c, (0x2742 bytes) 2024-04-24T00:34:14.315 INFO:teuthology.orchestra.run.smithi064.stdout:773 write 0x89e0e6f thru 0x89eea3d (0xdbcf bytes) 2024-04-24T00:34:14.326 INFO:teuthology.orchestra.run.smithi064.stdout:774 read 0x77df9a3 thru 0x77e29a9 (0x3007 bytes) 2024-04-24T00:34:14.326 INFO:teuthology.orchestra.run.smithi064.stdout:776 punch from 0x244c507 to 0x24576da, (0xb1d3 bytes) 2024-04-24T00:34:14.326 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:14.326 INFO:teuthology.orchestra.run.smithi064.stdout:778 read 0x9d2fdeb thru 0x9d30f75 (0x118b bytes) 2024-04-24T00:34:14.326 INFO:teuthology.orchestra.run.smithi064.stdout:779 write 0x10d15ef thru 0x10d59bf (0x43d1 bytes) 2024-04-24T00:34:14.329 INFO:teuthology.orchestra.run.smithi064.stdout:781 read 0x2dce816 thru 0x2dd34fd (0x4ce8 bytes) 2024-04-24T00:34:14.329 INFO:teuthology.orchestra.run.smithi064.stdout:783 read 0x49c5604 thru 0x49ca867 (0x5264 bytes) 2024-04-24T00:34:14.329 INFO:teuthology.orchestra.run.smithi064.stdout:785 punch from 0x1bb7e6d to 0x1bb87eb, (0x97e bytes) 2024-04-24T00:34:14.329 INFO:teuthology.orchestra.run.smithi064.stdout:787 read 0xa1be0e thru 0xa1f16c (0x335f bytes) 2024-04-24T00:34:14.329 INFO:teuthology.orchestra.run.smithi064.stdout:788 trunc from 0xb9ef230 to 0xbe3d395 2024-04-24T00:34:14.338 INFO:teuthology.orchestra.run.smithi064.stdout:790 clone 5 order 24 su 131072 sc 12 2024-04-24T00:34:14.341 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:14.336+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2036 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8843211909999997, 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020639231000000001, 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2036 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528464+0000", 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600000000000004e-07 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528467+0000", 2024-04-24T00:34:14.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8040000000000002e-06 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528474+0000", 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7270000000000007e-06 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528480+0000", 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3340000000000001e-06 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548902+0000", 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020421963000000001 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548991+0000", 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9332000000000005e-05 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549102+0000", 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111535 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2053 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8840610200000008, 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616257999999998, 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2053 2024-04-24T00:34:14.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528724+0000", 2024-04-24T00:34:14.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6400000000000002e-07 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528727+0000", 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6749999999999998e-06 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528734+0000", 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4739999999999999e-06 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528740+0000", 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4759999999999997e-06 2024-04-24T00:34:14.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549138+0000", 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020398315 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549225+0000", 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6802e-05 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549340+0000", 2024-04-24T00:34:14.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011495200000000001 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2054 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8840371539999996, 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020836107, 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2054 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528748+0000", 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999999e-07 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528750+0000", 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528756+0000", 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1249999999999998e-06 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528761+0000", 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549377+0000", 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616254000000001 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549468+0000", 2024-04-24T00:34:14.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1199000000000006e-05 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549583+0000", 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011484099999999999 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2055 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:14.430 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8840161739999992, 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022112625, 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2055 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:14.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528769+0000", 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9299999999999999e-07 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528771+0000", 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2809999999999998e-06 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528777+0000", 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.127e-06 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528782+0000", 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8330000000000001e-06 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550648+0000", 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021866450999999999 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550755+0000", 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010676800000000001 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550881+0000", 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012577199999999999 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2056 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8839944559999999, 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022390911999999999, 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2056 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528792+0000", 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1390000000000002e-06 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528798+0000", 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9869999999999996e-06 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528804+0000", 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4949999999999999e-06 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550928+0000", 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022124061 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551050+0000", 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121981 2024-04-24T00:34:14.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551181+0000", 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130858 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8839735110000007, 2024-04-24T00:34:14.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:34:14.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8839495970000009, 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8839224229999996, 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:34:14.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8838963179999997, 2024-04-24T00:34:14.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:34:14.456 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8834853369999998, 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.457 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8834655280000003, 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:14.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8834124689999996, 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:34:14.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8833776130000004, 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:34:14.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.475 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.477 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:14.477 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8833560489999996, 2024-04-24T00:34:14.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:34:14.477 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.477 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.477 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:14.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8833265570000002, 2024-04-24T00:34:14.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:34:14.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.485 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8833056950000007, 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:34:14.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:34:14.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8832831209999998, 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:34:14.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:34:14.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.201023623, 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:34:14.494 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:14.500 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.2006727079999999, 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:34:14.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.1937843880000001, 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:34:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:14.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:34:14.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:34:14.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:14.510 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:14.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:14.512 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:14.507+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:14.612 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:14.607+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10859 3.3 3:c6acc687:::rbd_object_map.115e8003c269:head [call rbd.object_map_load in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.649425+0000", 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.000220511, 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028435800000000002, 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:14.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10859 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.649425+0000", 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.649425+0000", 2024-04-24T00:34:14.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.649427+0000", 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.897e-06 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.649437+0000", 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8800000000000003e-06 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.649461+0000", 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4119999999999999e-05 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.649474+0000", 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2693e-05 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.649539+0000", 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4950000000000007e-05 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:14.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.649588+0000", 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9199000000000001e-05 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:34:14.656 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:14.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:14.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:14.713 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:14.708+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:14.788 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:14.788 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:14.788 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:14.788 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:14.788 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:14.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:14.814 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:14.811+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:14.841 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:14.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:14.858 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:14.858 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:14.876 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:14.877 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:14.898 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:14.898 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:14.915 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:14.912+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:14.919 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:14.919 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:14.939 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:14.940 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:14.963 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:14.964 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:14.976 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:14.976 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:14.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:15.025 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:15.023+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:15.117 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:15.115+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:15.152 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:15.152 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:15.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:15.218 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:15.212+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:15.288 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:15.288 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:15.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:15.343 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:15.340+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:15.419 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:15.415+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:15.466 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:15.466 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:15.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:15.520 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:15.519+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:15.601 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:15.601 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:15.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:15.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:34:15.659 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:15.656+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:15.711 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:15.711 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:15.711 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-24T00:34:15.711 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.339281 2024-04-24T00:34:15.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.33928141334187223 2024-04-24T00:34:15.723 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:15.720+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:15.767 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:15.767 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:15.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:15.831 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:15.828+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:15.924 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:15.920+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:15.974 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:15.974 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:15.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:16.025 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:16.023+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:16.126 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:16.123+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:16.148 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:16.149 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:16.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:16.226 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:16.223+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:16.280 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:16.280 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:16.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:16.327 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:16.323+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:16.415 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:16.416 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:16.428 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:16.424+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:16.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:16.530 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:16.527+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:16.564 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:16.564 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:16.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:16.623 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:34:16.634 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.3 primary-affinity to 0.339281 (8222352) 2024-04-24T00:34:16.634 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:16.631+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:16.731 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:16.728+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:16.768 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:16.768 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:16.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:16.832 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:16.828+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:16.899 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:16.899 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:16.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:16.933 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:16.928+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:16.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:17.033 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:17.031+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:17.034 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:17.034 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:17.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:17.134 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:17.135 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:17.135 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:17.131+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:17.151 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:17.151 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:17.168 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:17.168 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:17.185 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:17.185 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:17.206 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:17.206 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:17.210 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:17.211 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:17.211 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:17.211 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:17.211 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:17.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:17.227 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:17.248 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:17.248 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:17.248 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:17.243+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:17.269 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:17.270 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:17.336 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:17.332+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:17.350 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2025 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1638400~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817996388999999, 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077913280000000001, 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2025 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530290+0000", 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530291+0000", 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3200000000000005e-07 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530294+0000", 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530303+0000", 2024-04-24T00:34:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8820000000000007e-06 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530310+0000", 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0199999999999997e-06 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.537775+0000", 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074650569999999998 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.537853+0000", 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7961999999999997e-05 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538082+0000", 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0002288 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2026 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1769472~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817967229000001, 2024-04-24T00:34:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081592609999999993, 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2026 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530319+0000", 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530320+0000", 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6700000000000003e-07 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530323+0000", 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7420000000000002e-06 2024-04-24T00:34:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530332+0000", 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6059999999999992e-06 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530340+0000", 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2620000000000001e-06 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538180+0000", 2024-04-24T00:34:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078402369999999999 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538308+0000", 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012834000000000001 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.538479+0000", 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000170507 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2027 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817937263999999, 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.009211136, 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2027 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530349+0000", 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530350+0000", 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7199999999999999e-07 2024-04-24T00:34:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530353+0000", 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3390000000000001e-06 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530362+0000", 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0489999999999993e-06 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530370+0000", 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5739999999999998e-06 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539401+0000", 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090311609999999994 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539466+0000", 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4880000000000002e-05 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539561+0000", 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4561000000000005e-05 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2028 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817906741, 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095022319999999993, 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2028 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530380+0000", 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7899999999999998e-07 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530384+0000", 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0280000000000001e-06 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530392+0000", 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7830000000000004e-06 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530400+0000", 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2080000000000003e-06 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539616+0000", 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092167310000000006 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539722+0000", 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010527900000000001 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539882+0000", 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000160624 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2045 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817876677999999, 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097538699999999996, 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2045 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530410+0000", 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530411+0000", 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530414+0000", 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2430000000000001e-06 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530423+0000", 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3030000000000005e-06 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530430+0000", 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.917e-06 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.539923+0000", 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0094929349999999992 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540027+0000", 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010355 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540164+0000", 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013730799999999999 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2046 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817846471999999, 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010030647, 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2046 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530440+0000", 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530441+0000", 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3200000000000005e-07 2024-04-24T00:34:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530444+0000", 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8109999999999999e-06 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530453+0000", 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3409999999999992e-06 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530460+0000", 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3719999999999999e-06 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540209+0000", 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097491539999999995 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540326+0000", 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001166 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540471+0000", 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014483700000000001 2024-04-24T00:34:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2047 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817815803, 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010295987, 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2047 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530472+0000", 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2799999999999995e-07 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530475+0000", 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0939999999999999e-06 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530485+0000", 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7969999999999995e-06 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530491+0000", 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8709999999999999e-06 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540514+0000", 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010022499000000001 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540626+0000", 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111981 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540767+0000", 2024-04-24T00:34:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014091700000000001 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2048 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817784992, 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010591068, 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2048 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2699999999999999e-07 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530520+0000", 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8165999999999999e-05 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530533+0000", 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2065999999999999e-05 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540812+0000", 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010266239 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540925+0000", 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112676 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541093+0000", 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016786500000000001 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2049 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817728749, 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010815047, 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2049 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530562+0000", 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4369999999999999e-06 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530572+0000", 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530579+0000", 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541138+0000", 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010558209000000001 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541239+0000", 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101483 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541373+0000", 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133769 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2050 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.8176974, 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011052364, 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2050 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530590+0000", 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6599999999999996e-07 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530593+0000", 2024-04-24T00:34:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.937e-06 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530602+0000", 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4360000000000008e-06 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530609+0000", 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0090000000000001e-06 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541411+0000", 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01080191 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541511+0000", 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9839000000000004e-05 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541642+0000", 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130667 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2051 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817668040000001, 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011318002000000001, 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2051 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2900000000000004e-07 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530622+0000", 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9490000000000001e-06 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530632+0000", 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999994e-06 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.131e-06 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541683+0000", 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011044629 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120383 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541937+0000", 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132941 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2052 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817638840000001, 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011583872, 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2052 2024-04-24T00:34:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.44e-07 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530658+0000", 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9850000000000001e-06 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530668+0000", 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530675+0000", 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6240000000000002e-06 2024-04-24T00:34:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541978+0000", 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011302921 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542100+0000", 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121593 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542232+0000", 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013176600000000001 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2077 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817601440000001, 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011842221, 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2077 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6000000000000003e-07 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.095e-06 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530699+0000", 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530706+0000", 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2429999999999999e-06 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542275+0000", 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011568746 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542386+0000", 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011176200000000001 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542527+0000", 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014114499999999999 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2078 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817571114, 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012081010999999999, 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2078 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2200000000000004e-07 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530719+0000", 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9349999999999999e-06 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530728+0000", 2024-04-24T00:34:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0860000000000008e-06 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530735+0000", 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2169999999999997e-06 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542568+0000", 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011832540000000001 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542667+0000", 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9000999999999996e-05 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542797+0000", 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012971 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2079 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817541801000001, 2024-04-24T00:34:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012334128999999999, 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2079 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530748+0000", 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9040000000000001e-06 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2159999999999995e-06 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530764+0000", 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9890000000000003e-06 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542839+0000", 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012074354000000001 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542945+0000", 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106443 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543079+0000", 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013369599999999999 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2080 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817513001, 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012588825999999999, 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2080 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530777+0000", 2024-04-24T00:34:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.022e-06 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9879999999999993e-06 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530793+0000", 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1409999999999999e-06 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543122+0000", 2024-04-24T00:34:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012328274 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543230+0000", 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108046 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543362+0000", 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013286799999999999 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2081 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817484115999999, 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012843515, 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2081 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1500000000000005e-07 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.819e-06 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530815+0000", 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3540000000000006e-06 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530835+0000", 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9983000000000002e-05 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543405+0000", 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012569739 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543513+0000", 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010821500000000001 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543646+0000", 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289000000000001 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2082 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817440900999999, 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013081859, 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2082 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530849+0000", 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0750000000000002e-06 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530859+0000", 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7079999999999992e-06 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530866+0000", 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4309999999999997e-06 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543689+0000", 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012822750000000001 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543796+0000", 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106526 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543928+0000", 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131846 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2083 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817410627999999, 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013453964000000001, 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2083 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530877+0000", 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530887+0000", 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186e-05 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530896+0000", 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5480000000000007e-06 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530904+0000", 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4599999999999997e-06 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543970+0000", 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013066067000000001 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544178+0000", 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020785000000000001 2024-04-24T00:34:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544330+0000", 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015230699999999999 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2084 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.817372804, 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907047000000001, 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2084 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3099999999999998e-07 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530922+0000", 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.498e-06 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530932+0000", 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530940+0000", 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4769999999999997e-06 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544421+0000", 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013481739 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544673+0000", 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025143600000000001 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544821+0000", 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014819 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.435 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:17.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:17.437 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:17.432+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:17.538 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:17.532+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:17.569 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:17.570 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:17.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:17.639 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:17.636+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:17.682 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-04-24T00:34:17.717 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:17.717 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:17.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:17.781 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:17.779+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:17.840 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:17.835+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2984 3.5 e33/21 3:a2b34f77:::rbd_data.115e5ff16b0c.000000000000001c:head v 33'37, mlcod=30'35)", 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.247767827000001, 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437279999999999, 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613506+0000", 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613507+0000", 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.63e-07 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613521+0000", 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4545999999999999e-05 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613540+0000", 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8963000000000001e-05 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613550+0000", 2024-04-24T00:34:17.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0298e-05 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613622+0000", 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1476999999999993e-05 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.613645+0000", 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2940999999999999e-05 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.615112+0000", 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014666729999999999 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.615150+0000", 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8367e-05 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2979 3.9 e33/21 3:9717bf35:::rbd_data.115e5ff16b0c.000000000000001b:head v 33'28, mlcod=30'26)", 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:17.868 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.039072486, 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053184019999999999, 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:17.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822203+0000", 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3909999999999999e-06 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822482+0000", 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027925399999999999 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822509+0000", 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.707e-05 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822521+0000", 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1779e-05 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822626+0000", 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105165 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822653+0000", 2024-04-24T00:34:17.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6713e-05 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827477+0000", 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048240949999999996 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827520+0000", 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2935000000000001e-05 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.872 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3748 3.1f e33/21 3:feb5d058:::rbd_data.115e5ff16b0c.0000000000000030:head v 33'28, mlcod=33'26)", 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.982284283, 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028352379999999999, 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:17.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878991+0000", 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879014+0000", 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2677e-05 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879033+0000", 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9361999999999999e-05 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879043+0000", 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2299999999999997e-06 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879110+0000", 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7385999999999995e-05 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879132+0000", 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2405e-05 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881782+0000", 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026490979999999999 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881825+0000", 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3460000000000001e-05 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3747 3.9 e33/21 3:965825a6:::rbd_data.115e5ff16b0c.0000000000000038:head v 33'29, mlcod=33'27)", 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.973746832, 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003408908, 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887528+0000", 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2240000000000001e-06 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887593+0000", 2024-04-24T00:34:17.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4362999999999993e-05 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887606+0000", 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887614+0000", 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2290000000000003e-06 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887680+0000", 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5896999999999996e-05 2024-04-24T00:34:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887704+0000", 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3516e-05 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890888+0000", 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031837860000000001 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890936+0000", 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8464000000000002e-05 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2164678529999993, 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:34:17.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:34:17.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10260 3.9 e40/21 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head v 40'55, mlcod=38'53)", 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6359178989999998, 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027588550000000002, 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225357+0000", 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3319999999999999e-06 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226434+0000", 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010764730000000001 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226460+0000", 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6478e-05 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226476+0000", 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5769000000000001e-05 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226662+0000", 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186202 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226706+0000", 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3760000000000001e-05 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.228025+0000", 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013191699999999999 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.228115+0000", 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9671000000000003e-05 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.888 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10336 3.9 e40/21 3:912611d7:::rbd_data.115e380d621c.0000000000000213:head v 40'56, mlcod=39'54)", 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4455660760000004, 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021373809999999998, 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415710+0000", 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.973e-06 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415797+0000", 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7194999999999994e-05 2024-04-24T00:34:17.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415825+0000", 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7501999999999999e-05 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415838+0000", 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3232e-05 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415929+0000", 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1360000000000006e-05 2024-04-24T00:34:17.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415962+0000", 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2570999999999997e-05 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.417692+0000", 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001729801 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.417845+0000", 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153747 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10361 3.1f e40/21 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head v 40'65, mlcod=40'63)", 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4050540219999998, 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020517270000000002, 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456223+0000", 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6970000000000002e-06 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456269+0000", 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.617e-05 2024-04-24T00:34:17.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456302+0000", 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3028000000000003e-05 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456323+0000", 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456451+0000", 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012818899999999999 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456512+0000", 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0680000000000002e-05 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458154+0000", 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016422330000000001 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458272+0000", 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117477 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.896 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10425 3.4 e41/21 3:2657577c:::rbd_data.115e380d621c.0000000000000210:head v 41'28, mlcod=39'26)", 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2429971789999996, 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041008809999999998, 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618285+0000", 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5580000000000003e-06 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618309+0000", 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-05 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618321+0000", 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.164e-05 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619328+0000", 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010071259999999999 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619360+0000", 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1581000000000001e-05 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622325+0000", 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029648610000000001 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622378+0000", 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3235000000000001e-05 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10415 3.1f e41/21 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head v 41'66, mlcod=40'64)", 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.241758463, 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029607800000000001, 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.53e-07 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619520+0000", 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3710000000000002e-06 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619530+0000", 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4609999999999997e-06 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619536+0000", 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2879999999999998e-06 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619618+0000", 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2091999999999999e-05 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619640+0000", 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2042e-05 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622396+0000", 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027557300000000001 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622476+0000", 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0543000000000005e-05 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10531 3.9 e41/21 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head v 41'60, mlcod=41'58)", 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.0816139890000001, 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030187019999999998, 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:17.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779662+0000", 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6819999999999999e-06 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779672+0000", 2024-04-24T00:34:17.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0482999999999999e-05 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779700+0000", 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8218999999999999e-05 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779713+0000", 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2123000000000001e-05 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779827+0000", 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114355 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779866+0000", 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9449000000000001e-05 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.782618+0000", 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027520980000000001 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.782679+0000", 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0293e-05 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10541 3.1f e41/21 3:f85c3e31:::rbd_data.115e380d621c.0000000000000141:head v 41'69, mlcod=41'67)", 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.0757964800000002, 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025351169999999999, 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785480+0000", 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2879999999999999e-06 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785493+0000", 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3036e-05 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785529+0000", 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6043e-05 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785546+0000", 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6897000000000001e-05 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785630+0000", 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4124999999999998e-05 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785657+0000", 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.724e-05 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.787959+0000", 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023013920000000002 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.788013+0000", 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4095999999999998e-05 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10584 3.1f e41/21 3:f85c3e31:::rbd_data.115e380d621c.0000000000000141:head v 41'71, mlcod=41'69)", 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.848796+0000", 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.0124778599999997, 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019548859999999999, 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848796+0000", 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848796+0000", 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848797+0000", 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.212e-06 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848812+0000", 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4834999999999999e-05 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848829+0000", 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6580999999999999e-05 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848837+0000", 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0530000000000007e-06 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848962+0000", 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012488399999999999 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.849033+0000", 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0970000000000007e-05 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.850689+0000", 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001656343 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.850751+0000", 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2008000000000007e-05 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10596 3.1f e41/21 3:ff376758:::rbd_data.115e380d621c.000000000000028b:head v 41'72, mlcod=41'70)", 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.926625+0000", 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9346490709999999, 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019602, 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926625+0000", 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926625+0000", 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926628+0000", 2024-04-24T00:34:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8629999999999999e-06 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926683+0000", 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.533e-05 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926715+0000", 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1498000000000002e-05 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926732+0000", 2024-04-24T00:34:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6869000000000001e-05 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926818+0000", 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6213999999999998e-05 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926842+0000", 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.425e-05 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.928534+0000", 2024-04-24T00:34:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016915350000000001 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.928585+0000", 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1641000000000002e-05 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10635 3.5 e41/21 3:a07ee5e3:::rbd_header.115e380d621c:head v 41'82, mlcod=41'80)", 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.020776+0000", 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8404979020000001, 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00169148, 2024-04-24T00:34:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.020776+0000", 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.020776+0000", 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.020778+0000", 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.691e-06 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.020803+0000", 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4728000000000001e-05 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.020822+0000", 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9885999999999999e-05 2024-04-24T00:34:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.020833+0000", 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0926e-05 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.020998+0000", 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016468599999999999 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.021073+0000", 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4622000000000005e-05 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.022386+0000", 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001313743 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.022468+0000", 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1198000000000004e-05 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10672 3.4 e41/21 3:20eab9c7:::rbd_data.115e380d621c.00000000000002b6:head v 41'29, mlcod=39'27)", 2024-04-24T00:34:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.087977+0000", 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7732966110000001, 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019443500000000001, 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.087977+0000", 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.087977+0000", 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.087979+0000", 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3990000000000001e-06 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.087992+0000", 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3286e-05 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.088014+0000", 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2022999999999999e-05 2024-04-24T00:34:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.088024+0000", 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0087e-05 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.088153+0000", 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128882 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.088226+0000", 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2484999999999995e-05 2024-04-24T00:34:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.089804+0000", 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015788009999999999 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.089922+0000", 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011738700000000001 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10682 3.9 e41/21 3:9048ec18:::rbd_data.115e380d621c.000000000000038f:head v 41'62, mlcod=41'60)", 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.110930+0000", 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7503437489999998, 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020353319999999999, 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110930+0000", 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110930+0000", 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110932+0000", 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.114e-06 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110958+0000", 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5707e-05 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110982+0000", 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4009e-05 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110996+0000", 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3389e-05 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111062+0000", 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6002999999999998e-05 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111086+0000", 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4983000000000001e-05 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.112928+0000", 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001841472 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.112966+0000", 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7654999999999998e-05 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10676 3.4 e41/21 3:20eab9c7:::rbd_data.115e380d621c.00000000000002b6:head v 41'30, mlcod=41'28)", 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.111498+0000", 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7497764949999999, 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021122799999999998, 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111498+0000", 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111498+0000", 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111498+0000", 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111505+0000", 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949999999999999e-06 2024-04-24T00:34:17.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111517+0000", 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1588e-05 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111524+0000", 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4880000000000001e-06 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111614+0000", 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9141000000000006e-05 2024-04-24T00:34:17.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111636+0000", 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2532999999999999e-05 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.113578+0000", 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019419509999999999 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.113610+0000", 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.171e-05 2024-04-24T00:34:17.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10705 3.9 e41/21 3:93f8bbf0:::rbd_data.115e380d621c.0000000000000399:head v 41'63, mlcod=41'61)", 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.699037154, 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034786589999999998, 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:17.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162238+0000", 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162284+0000", 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6122999999999999e-05 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162296+0000", 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1871e-05 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162304+0000", 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7030000000000006e-06 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162390+0000", 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6582999999999994e-05 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162431+0000", 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0577999999999997e-05 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165655+0000", 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003224612 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165716+0000", 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0251999999999998e-05 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10758 3.9 e41/21 3:93c11b29:::rbd_data.115e380d621c.0000000000000042:head v 41'65, mlcod=41'63)", 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.266288+0000", 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.594986171, 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034766659999999998, 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:17.942 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266288+0000", 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266288+0000", 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266290+0000", 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9e-06 2024-04-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266342+0000", 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1823000000000002e-05 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266377+0000", 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5148000000000003e-05 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266392+0000", 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5325999999999999e-05 2024-04-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266548+0000", 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015587500000000001 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266616+0000", 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8516000000000006e-05 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.269634+0000", 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003017572 2024-04-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.269765+0000", 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013050599999999999 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:17.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:17.948 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:17.943+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:18.049 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:18.047+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:18.119 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:18.119 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:18.119 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:18.119 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:18.124 INFO:teuthology.orchestra.run.smithi064.stdout:792 trunc from 0xbe3d395 to 0x5c24ca3 2024-04-24T00:34:18.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:18.171 INFO:teuthology.orchestra.run.smithi064.stdout:793 write 0x5f0d994 thru 0x5f14e50 (0x74bd bytes) 2024-04-24T00:34:18.172 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:18.167+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:18.174 INFO:teuthology.orchestra.run.smithi064.stdout:794 read 0x3be9011 thru 0x3bf204e (0x903e bytes) 2024-04-24T00:34:18.198 INFO:teuthology.orchestra.run.smithi064.stdout:795 writesame 0x962a5e2 thru 0x9633daf data_size 0x33(0x97ce bytes) 2024-04-24T00:34:18.201 INFO:teuthology.orchestra.run.smithi064.stdout:796 read 0xd7d972 thru 0xd80b81 (0x3210 bytes) 2024-04-24T00:34:18.202 INFO:teuthology.orchestra.run.smithi064.stdout:798 punch from 0xbf591f to 0xc03dd1, (0xe4b2 bytes) 2024-04-24T00:34:18.202 INFO:teuthology.orchestra.run.smithi064.stdout:799 punch from 0x511e07c to 0x511eb15, (0xa99 bytes) 2024-04-24T00:34:18.202 INFO:teuthology.orchestra.run.smithi064.stdout:801 read 0x296da3b thru 0x2975e05 (0x83cb bytes) 2024-04-24T00:34:18.203 INFO:teuthology.orchestra.run.smithi064.stdout:802 writesame 0xd55aff thru 0xd5a971 data_size 0x13(0x4e73 bytes) 2024-04-24T00:34:18.207 INFO:teuthology.orchestra.run.smithi064.stdout:803 read 0x5e32462 thru 0x5e3b160 (0x8cff bytes) 2024-04-24T00:34:18.208 INFO:teuthology.orchestra.run.smithi064.stdout:804 writesame 0x58fe1cc thru 0x5907d95 data_size 0x66(0x9bca bytes) 2024-04-24T00:34:18.232 INFO:teuthology.orchestra.run.smithi064.stdout:805 write 0xc9d2e90 thru 0xc9d5bfb (0x2d6c bytes) 2024-04-24T00:34:18.239 INFO:teuthology.orchestra.run.smithi064.stdout:806 read 0x8d862dd thru 0x8d8bfe1 (0x5d05 bytes) 2024-04-24T00:34:18.240 INFO:teuthology.orchestra.run.smithi064.stdout:807 writesame 0xdf3344 thru 0xe0060f data_size 0x12(0xd2cc bytes) 2024-04-24T00:34:18.246 INFO:teuthology.orchestra.run.smithi064.stdout:808 read 0x2caee49 thru 0x2cbc5d1 (0xd789 bytes) 2024-04-24T00:34:18.247 INFO:teuthology.orchestra.run.smithi064.stdout:809 punch from 0x2068bb6 to 0x206994e, (0xd98 bytes) 2024-04-24T00:34:18.247 INFO:teuthology.orchestra.run.smithi064.stdout:810 write 0x1efe3e thru 0x1f8eeb (0x90ae bytes) 2024-04-24T00:34:18.251 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:18.247+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:18.272 INFO:teuthology.orchestra.run.smithi064.stdout:811 trunc from 0xc9d5bfc to 0x4deddb6 2024-04-24T00:34:18.301 INFO:teuthology.orchestra.run.smithi064.stdout:812 read 0x18833c1 thru 0x18925e3 (0xf223 bytes) 2024-04-24T00:34:18.301 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:18.302 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:18.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:18.323 INFO:teuthology.orchestra.run.smithi064.stdout:813 write 0xa2e9593 thru 0xa2ef95e (0x63cc bytes) 2024-04-24T00:34:18.327 INFO:teuthology.orchestra.run.smithi064.stdout:814 punch from 0x3a9d4a9 to 0x3aab5d3, (0xe12a bytes) 2024-04-24T00:34:18.330 INFO:teuthology.orchestra.run.smithi064.stdout:815 trunc from 0xa2ef95f to 0xcb4d2f7 2024-04-24T00:34:18.342 INFO:teuthology.orchestra.run.smithi064.stdout:816 writesame 0x2d0da87 thru 0x2d1b214 data_size 0x11(0xd78e bytes) 2024-04-24T00:34:18.351 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:18.347+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:18.356 INFO:teuthology.orchestra.run.smithi064.stdout:818 read 0x7e456a2 thru 0x7e4d25e (0x7bbd bytes) 2024-04-24T00:34:18.357 INFO:teuthology.orchestra.run.smithi064.stdout:819 punch from 0x9d0905a to 0x9d12c02, (0x9ba8 bytes) 2024-04-24T00:34:18.357 INFO:teuthology.orchestra.run.smithi064.stdout:820 write 0x7152d5a thru 0x715be16 (0x90bd bytes) 2024-04-24T00:34:18.367 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.367 INFO:teuthology.orchestra.run.smithi064.stdout:822 write 0x4b8c9e7 thru 0x4b8e4f6 (0x1b10 bytes) 2024-04-24T00:34:18.367 INFO:teuthology.orchestra.run.smithi064.stdout:823 writesame 0x6cbed4a thru 0x6cc115d data_size 0x2(0x2414 bytes) 2024-04-24T00:34:18.371 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0xeb2a54c 2024-04-24T00:34:18.375 INFO:teuthology.orchestra.run.smithi064.stdout:824 trunc from 0xcb4d2f7 to 0xeb2a54c 2024-04-24T00:34:18.387 INFO:teuthology.orchestra.run.smithi064.stdout:825 read 0x2819872 thru 0x28261ac (0xc93b bytes) 2024-04-24T00:34:18.388 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.388 INFO:teuthology.orchestra.run.smithi064.stdout:829 punch from 0xb1b30 to 0xb1fe0, (0x4b0 bytes) 2024-04-24T00:34:18.388 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.388 INFO:teuthology.orchestra.run.smithi064.stdout:832 read 0x4e1192b thru 0x4e17ee0 (0x65b6 bytes) 2024-04-24T00:34:18.389 INFO:teuthology.orchestra.run.smithi064.stdout:833 read 0x47040a7 thru 0x471309e (0xeff8 bytes) 2024-04-24T00:34:18.390 INFO:teuthology.orchestra.run.smithi064.stdout:834 write 0x59a646 thru 0x59a656 (0x11 bytes) 2024-04-24T00:34:18.394 INFO:teuthology.orchestra.run.smithi064.stdout:835 read 0x1ba1106 thru 0x1bada02 (0xc8fd bytes) 2024-04-24T00:34:18.405 INFO:teuthology.orchestra.run.smithi064.stdout:836 trunc from 0xeb2a54c to 0x9055f57 2024-04-24T00:34:18.423 INFO:teuthology.orchestra.run.smithi064.stdout:837 write 0x4f4bda1 thru 0x4f4c467 (0x6c7 bytes) 2024-04-24T00:34:18.425 INFO:teuthology.orchestra.run.smithi064.stdout:838 write 0x42365b3 thru 0x4245eb4 (0xf902 bytes) 2024-04-24T00:34:18.452 INFO:teuthology.orchestra.run.smithi064.stdout:839 write 0xbc7a392 thru 0xbc7cf13 (0x2b82 bytes) 2024-04-24T00:34:18.452 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:18.448+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:18.454 INFO:teuthology.orchestra.run.smithi064.stdout:842 write 0x1537fab thru 0x153925e (0x12b4 bytes) 2024-04-24T00:34:18.457 INFO:teuthology.orchestra.run.smithi064.stdout:846 writesame 0xeb26b1 thru 0xeb2fe4 data_size 0x4(0x934 bytes) 2024-04-24T00:34:18.464 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:18.464 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2036 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931865147, 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020639231000000001, 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2036 2024-04-24T00:34:18.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528464+0000", 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600000000000004e-07 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528467+0000", 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8040000000000002e-06 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528474+0000", 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7270000000000007e-06 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528480+0000", 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3340000000000001e-06 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548902+0000", 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020421963000000001 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548991+0000", 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9332000000000005e-05 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549102+0000", 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111535 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2053 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931604975999999, 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616257999999998, 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2053 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528724+0000", 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6400000000000002e-07 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528727+0000", 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6749999999999998e-06 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528734+0000", 2024-04-24T00:34:18.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4739999999999999e-06 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528740+0000", 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4759999999999997e-06 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549138+0000", 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020398315 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549225+0000", 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6802e-05 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549340+0000", 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011495200000000001 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2054 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:18.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.93158111, 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020836107, 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2054 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528748+0000", 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999999e-07 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528750+0000", 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-04-24T00:34:18.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528756+0000", 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1249999999999998e-06 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528761+0000", 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549377+0000", 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616254000000001 2024-04-24T00:34:18.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549468+0000", 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1199000000000006e-05 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549583+0000", 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011484099999999999 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2055 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931560129999999, 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022112625, 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2055 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.478 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528769+0000", 2024-04-24T00:34:18.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9299999999999999e-07 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528771+0000", 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2809999999999998e-06 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528777+0000", 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.127e-06 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528782+0000", 2024-04-24T00:34:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8330000000000001e-06 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550648+0000", 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021866450999999999 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550755+0000", 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010676800000000001 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550881+0000", 2024-04-24T00:34:18.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012577199999999999 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2056 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931538412, 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022390911999999999, 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2056 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528792+0000", 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1390000000000002e-06 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528798+0000", 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9869999999999996e-06 2024-04-24T00:34:18.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528804+0000", 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4949999999999999e-06 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550928+0000", 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022124061 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551050+0000", 2024-04-24T00:34:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121981 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551181+0000", 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130858 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931517467000001, 2024-04-24T00:34:18.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931493552999999, 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:34:18.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931466379, 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931440274, 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:18.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:34:18.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:34:18.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931029293, 2024-04-24T00:34:18.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:34:18.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:34:18.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.931009484, 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.508 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:18.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.930956425, 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:34:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.930921569000001, 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.930900005, 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.930870513, 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.930849650999999, 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.930827077, 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2485675790000004, 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:34:18.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:34:18.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2482166640000001, 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:34:18.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:34:18.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2413283440000002, 2024-04-24T00:34:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:18.554 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:18.555 INFO:teuthology.orchestra.run.smithi064.stdout:847 write 0xc8fb892 thru 0xc904bfa (0x9369 bytes) 2024-04-24T00:34:18.556 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:18.557 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.557 INFO:teuthology.orchestra.run.smithi064.stdout:849 write 0x8a0d355 thru 0x8a13253 (0x5eff bytes) 2024-04-24T00:34:18.557 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.557 INFO:teuthology.orchestra.run.smithi064.stdout:851 read 0x3b49a3f thru 0x3b52008 (0x85ca bytes) 2024-04-24T00:34:18.557 INFO:teuthology.orchestra.run.smithi064.stdout:852 write 0x247d9f5 thru 0x247f3d8 (0x19e4 bytes) 2024-04-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi064.stdout:853 trunc from 0xc904bfb to 0xac63265 2024-04-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi064.stdout:854 writesame 0x1b4ec16 thru 0x1b57dd0 data_size 0x1(0x91bb bytes) 2024-04-24T00:34:18.558 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:18.552+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:18.656 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:18.651+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:18.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:34:18.659 INFO:teuthology.orchestra.run.smithi064.stdout:855 write 0x7bf474d thru 0x7bfa000 (0x58b4 bytes) 2024-04-24T00:34:18.667 INFO:teuthology.orchestra.run.smithi064.stdout:856 trunc from 0xac63265 to 0x9868bbb 2024-04-24T00:34:18.687 INFO:teuthology.orchestra.run.smithi064.stdout:857 read 0x62f7a9 thru 0x634a7f (0x52d7 bytes) 2024-04-24T00:34:18.688 INFO:teuthology.orchestra.run.smithi064.stdout:858 punch from 0x49e0da to 0x4a870d, (0xa633 bytes) 2024-04-24T00:34:18.693 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:18.693 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:18.693 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:18.694 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:18.706 INFO:teuthology.orchestra.run.smithi064.stdout:859 writesame 0xbef2858 thru 0xbefaacd data_size 0x1(0x8276 bytes) 2024-04-24T00:34:18.709 INFO:teuthology.orchestra.run.smithi064.stdout:861 punch from 0x93aa8e7 to 0x93b6f0a, (0xc623 bytes) 2024-04-24T00:34:18.709 INFO:teuthology.orchestra.run.smithi064.stdout:862 read 0x9e9f142 thru 0x9eae0f4 (0xefb3 bytes) 2024-04-24T00:34:18.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:18.713 INFO:teuthology.orchestra.run.smithi064.stdout:863 trunc from 0xbefaace to 0xdd9ef30 2024-04-24T00:34:18.725 INFO:teuthology.orchestra.run.smithi064.stdout:864 write 0x81940ac thru 0x8198c85 (0x4bda bytes) 2024-04-24T00:34:18.727 INFO:teuthology.orchestra.run.smithi064.stdout:865 read 0x2b7e654 thru 0x2b854eb (0x6e98 bytes) 2024-04-24T00:34:18.728 INFO:teuthology.orchestra.run.smithi064.stdout:866 punch from 0xc079f2b to 0xc07db77, (0x3c4c bytes) 2024-04-24T00:34:18.728 INFO:teuthology.orchestra.run.smithi064.stdout:867 read 0x3732e89 thru 0x373b20c (0x8384 bytes) 2024-04-24T00:34:18.739 INFO:teuthology.orchestra.run.smithi064.stdout:868 trunc from 0xdd9ef30 to 0x720c0ad 2024-04-24T00:34:18.751 INFO:teuthology.orchestra.run.smithi064.stdout:869 read 0x5d05c54 thru 0x5d07b68 (0x1f15 bytes) 2024-04-24T00:34:18.757 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:18.752+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:18.760 INFO:teuthology.orchestra.run.smithi064.stdout:870 trunc from 0x720c0ad to 0xd5615e7 2024-04-24T00:34:18.779 INFO:teuthology.orchestra.run.smithi064.stdout:871 trunc from 0xd5615e7 to 0x5e2c073 2024-04-24T00:34:18.791 INFO:teuthology.orchestra.run.smithi064.stdout:872 read 0x2762a2a thru 0x27644dd (0x1ab4 bytes) 2024-04-24T00:34:18.791 INFO:teuthology.orchestra.run.smithi064.stdout:874 write 0x208f528 thru 0x20931b6 (0x3c8f bytes) 2024-04-24T00:34:18.796 INFO:teuthology.orchestra.run.smithi064.stdout:877 trunc from 0x5e2c073 to 0x4adfd88 2024-04-24T00:34:18.826 INFO:teuthology.orchestra.run.smithi064.stdout:878 writesame 0xbcc0ea4 thru 0xbccb7b6 data_size 0x1(0xa913 bytes) 2024-04-24T00:34:18.827 INFO:teuthology.orchestra.run.smithi064.stdout:879 write 0x8bd06 thru 0x949e7 (0x8ce2 bytes) 2024-04-24T00:34:18.834 INFO:teuthology.orchestra.run.smithi064.stdout:880 trunc from 0xbccb7b7 to 0xdc7e90e 2024-04-24T00:34:18.841 INFO:teuthology.orchestra.run.smithi064.stdout:881 write 0x931828b thru 0x931d196 (0x4f0c bytes) 2024-04-24T00:34:18.842 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:18.842 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:18.842 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:18.842 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:18.842 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:18.845 INFO:teuthology.orchestra.run.smithi064.stdout:883 read 0x5bcbfc6 thru 0x5bd1cfe (0x5d39 bytes) 2024-04-24T00:34:18.846 INFO:teuthology.orchestra.run.smithi064.stdout:885 punch from 0xd4c8094 to 0xd4d30ed, (0xb059 bytes) 2024-04-24T00:34:18.846 INFO:teuthology.orchestra.run.smithi064.stdout:886 read 0xab49eaf thru 0xab4c85b (0x29ad bytes) 2024-04-24T00:34:18.846 INFO:teuthology.orchestra.run.smithi064.stdout:888 writesame 0x48bf2f4 thru 0x48cca86 data_size 0x1d(0xd793 bytes) 2024-04-24T00:34:18.849 INFO:teuthology.orchestra.run.smithi064.stdout:889 writesame 0x48b2975 thru 0x48bc194 data_size 0x10(0x9820 bytes) 2024-04-24T00:34:18.856 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.856 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.856 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.856 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.857 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:18.855+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:18.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:18.870 INFO:teuthology.orchestra.run.smithi064.stdout:895 trunc from 0xdc7e90e to 0x64ab78f 2024-04-24T00:34:18.885 INFO:teuthology.orchestra.run.smithi064.stdout:896 read 0x2573802 thru 0x2574835 (0x1034 bytes) 2024-04-24T00:34:18.885 INFO:teuthology.orchestra.run.smithi064.stdout:897 punch from 0x5129eaa to 0x5134b6b, (0xacc1 bytes) 2024-04-24T00:34:18.885 INFO:teuthology.orchestra.run.smithi064.stdout:898 read 0x381576f thru 0x381d0e0 (0x7972 bytes) 2024-04-24T00:34:18.909 INFO:teuthology.orchestra.run.smithi064.stdout:899 writesame 0xdddcc0e thru 0xdddf5ad data_size 0x20(0x29a0 bytes) 2024-04-24T00:34:18.909 INFO:teuthology.orchestra.run.smithi064.stdout:901 writesame 0x58cf4b1 thru 0x58dcf62 data_size 0x5d(0xdab2 bytes) 2024-04-24T00:34:18.912 INFO:teuthology.orchestra.run.smithi064.stdout:902 write 0xaaa4744 thru 0xaab307a (0xe937 bytes) 2024-04-24T00:34:18.918 INFO:teuthology.orchestra.run.smithi064.stdout:903 punch from 0xd1c6afd to 0xd1d47e2, (0xdce5 bytes) 2024-04-24T00:34:18.918 INFO:teuthology.orchestra.run.smithi064.stdout:905 read 0x5b4f75 thru 0x5bd397 (0x8423 bytes) 2024-04-24T00:34:18.919 INFO:teuthology.orchestra.run.smithi064.stdout:906 punch from 0x7076608 to 0x707750d, (0xf05 bytes) 2024-04-24T00:34:18.919 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:18.919 INFO:teuthology.orchestra.run.smithi064.stdout:908 write 0x7f20b3d thru 0x7f27584 (0x6a48 bytes) 2024-04-24T00:34:18.922 INFO:teuthology.orchestra.run.smithi064.stdout:910 punch from 0xcd3bf6 to 0xcd5047, (0x1451 bytes) 2024-04-24T00:34:18.922 INFO:teuthology.orchestra.run.smithi064.stdout:911 write 0x4281ec3 thru 0x429196b (0xfaa9 bytes) 2024-04-24T00:34:18.928 INFO:teuthology.orchestra.run.smithi064.stdout:912 clone 6 order 22 su 65536 sc 14 2024-04-24T00:34:18.959 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:18.955+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:19.030 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:19.030 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:19.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3338 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13434880~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.196296709, 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021864103999999999, 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3338 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.031 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3699999999999999e-07 2024-04-24T00:34:19.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828608+0000", 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.216e-06 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828619+0000", 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.172e-05 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828627+0000", 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.79e-06 2024-04-24T00:34:19.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850192+0000", 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021564967000000001 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850314+0000", 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121789 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850467+0000", 2024-04-24T00:34:19.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015268500000000001 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3339 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13565952~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.196261046, 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022319400999999999, 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3339 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828639+0000", 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0100000000000001e-06 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828643+0000", 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2300000000000002e-06 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828652+0000", 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000001e-06 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828659+0000", 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850539+0000", 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021879770999999999 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850647+0000", 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010767 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850958+0000", 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031074100000000001 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.039 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3400 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13828096~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.189464625999999, 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022009407000000002, 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3400 2024-04-24T00:34:19.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835436+0000", 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8139999999999999e-06 2024-04-24T00:34:19.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835462+0000", 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5273e-05 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835489+0000", 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6859000000000002e-05 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835507+0000", 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8144000000000001e-05 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.856821+0000", 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021313959 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857070+0000", 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024914099999999998 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857444+0000", 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037421699999999998 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3401 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13959168~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.18936815, 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022623199, 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3401 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835532+0000", 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3769999999999999e-06 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835540+0000", 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2969999999999998e-06 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835559+0000", 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9199999999999999e-05 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835574+0000", 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4983e-05 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857648+0000", 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022074522999999999 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857816+0000", 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016745200000000001 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858154+0000", 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033836700000000001 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3402 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14090240~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.18930529, 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023133308000000002, 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3402 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835595+0000", 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.043e-06 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835602+0000", 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7329999999999997e-06 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835624+0000", 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2079999999999999e-05 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835638+0000", 2024-04-24T00:34:19.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4270000000000001e-05 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858219+0000", 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022580586 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858442+0000", 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000223191 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858727+0000", 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028540500000000002 2024-04-24T00:34:19.052 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3403 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14221312~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.189241874, 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023734209999999999, 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3403 2024-04-24T00:34:19.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835658+0000", 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.083e-06 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835664+0000", 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5999999999999997e-06 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835683+0000", 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8604e-05 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835697+0000", 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4576000000000001e-05 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858838+0000", 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023140524999999999 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859063+0000", 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000225598 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859392+0000", 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032822400000000001 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3404 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14352384~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.189183325, 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024301263, 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3404 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:19.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835717+0000", 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835760+0000", 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2993e-05 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835781+0000", 2024-04-24T00:34:19.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0743000000000001e-05 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835796+0000", 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5233e-05 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859493+0000", 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023697296999999999 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859722+0000", 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000228619 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860017+0000", 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029529399999999998 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.061 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3405 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14483456~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.189079975, 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024591302999999998, 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3405 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835821+0000", 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835827+0000", 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5300000000000002e-06 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835845+0000", 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8096000000000001e-05 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835859+0000", 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3916e-05 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860117+0000", 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024257616999999999 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860241+0000", 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012400200000000001 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860411+0000", 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169562 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.065 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3406 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14614528~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.189019666, 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024976411, 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.066 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3406 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835881+0000", 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835887+0000", 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3369999999999996e-06 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835905+0000", 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7592999999999999e-05 2024-04-24T00:34:19.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835919+0000", 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4277000000000001e-05 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860500+0000", 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024580722999999999 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860651+0000", 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015075900000000001 2024-04-24T00:34:19.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860856+0000", 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020540200000000001 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3455 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.186200144000001, 2024-04-24T00:34:19.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022775208000000002, 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3455 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838702+0000", 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838715+0000", 2024-04-24T00:34:19.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2959000000000001e-05 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838740+0000", 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4893e-05 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838753+0000", 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3176999999999999e-05 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860986+0000", 2024-04-24T00:34:19.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022233541999999999 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.861203+0000", 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000216472 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.861474+0000", 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027145799999999997 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.074 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.384161188, 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:34:19.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:34:19.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.384084989, 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:34:19.079 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:19.083 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.383626815, 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:19.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.383569508000001, 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:34:19.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.091 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.383310542, 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.092 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:34:19.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:34:19.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.38238907, 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:34:19.096 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:34:19.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:19.100 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.8105743399999996, 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:34:19.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:34:19.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.106 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.7950450910000004, 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:34:19.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:34:19.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.1663217709999998, 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:34:19.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:34:19.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:34:19.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:19.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.1512866439999998, 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:34:19.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:19.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:19.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:19.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:34:19.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:34:19.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.121 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:19.121 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:19.121 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:19.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:19.123 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:19.119+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:19.224 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:19.220+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:19.251 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:19.251 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:19.251 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:19.251 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:19.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:19.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:19.333 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:19.331+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:19.426 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:19.420+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:19.429 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:19.429 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:19.429 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:19.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:19.444 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:19.445 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:19.461 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:19.462 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:19.478 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:19.478 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:19.495 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:19.495 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:19.515 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:19.531 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:19.528+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:19.533 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:19.551 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:19.552 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:19.572 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:19.572 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:19.605 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:19.605 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:19.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:19.662 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:34:19.663 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:19.660+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:19.728 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:19.724+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:19.788 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:19.788 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:19.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:19.829 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:19.827+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:19.930 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:19.927+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:19.965 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:19.965 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:19.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:20.030 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:20.027+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:20.103 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:20.103 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:20.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:20.163 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:20.160+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:20.232 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:20.228+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:20.241 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:20.241 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:20.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:20.333 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:20.331+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:20.424 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:20.424 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:20.434 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:20.428+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:20.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:20.534 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:20.528+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:20.606 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:20.606 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:20.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:20.665 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:20.663+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:20.736 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:20.732+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:20.776 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:20.777 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:20.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:20.877 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:20.875+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:20.938 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:20.935+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:20.994 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:20.994 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:21.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:21.049 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:21.047+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:21.140 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:21.136+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:21.173 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:21.173 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:21.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:21.241 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:21.239+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:21.307 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:21.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:21.342 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:21.336+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:21.443 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:21.440+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:21.493 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:21.493 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:21.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:21.544 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:21.540+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:21.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:21.634 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:21.635 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:21.645 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:21.643+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:21.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:21.663 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:21.664 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:21.664 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-24T00:34:21.664 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:34:21.664 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:34:21.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-04-24T00:34:21.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:34:21.746 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:21.743+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:21.778 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:21.778 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:21.814 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:21.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:21.827 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:21.827 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:21.840 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:21.840 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:21.847 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:21.844+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:21.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:21.855 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:21.855 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T00:34:21.855 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:21.861 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:21.861 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:21.871 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-04-24T00:34:21.885 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:21.922 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:21.943 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:21.943 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:21.948 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:21.943+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:21.976 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:21.976 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:21.992 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:21.992 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:21.993 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:21.993 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:21.993 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:22.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:22.061 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:22.059+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:22.150 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:22.144+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:22.182 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2979 3.9 e33/21 3:9717bf35:::rbd_data.115e5ff16b0c.000000000000001b:head v 33'28, mlcod=30'26)", 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.355876632000001, 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053184019999999999, 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822203+0000", 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3909999999999999e-06 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822482+0000", 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027925399999999999 2024-04-24T00:34:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822509+0000", 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.707e-05 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822521+0000", 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1779e-05 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822626+0000", 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105165 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822653+0000", 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6713e-05 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827477+0000", 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048240949999999996 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827520+0000", 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2935000000000001e-05 2024-04-24T00:34:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3748 3.1f e33/21 3:feb5d058:::rbd_data.115e5ff16b0c.0000000000000030:head v 33'28, mlcod=33'26)", 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.299088429000001, 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028352379999999999, 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878990+0000", 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.878991+0000", 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879014+0000", 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2677e-05 2024-04-24T00:34:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879033+0000", 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9361999999999999e-05 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879043+0000", 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2299999999999997e-06 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879110+0000", 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7385999999999995e-05 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.879132+0000", 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2405e-05 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881782+0000", 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026490979999999999 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.881825+0000", 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3460000000000001e-05 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:3747 3.9 e33/21 3:965825a6:::rbd_data.115e5ff16b0c.0000000000000038:head v 33'29, mlcod=33'27)", 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.290550977999999, 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003408908, 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887527+0000", 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887528+0000", 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2240000000000001e-06 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887593+0000", 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4362999999999993e-05 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887606+0000", 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887614+0000", 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2290000000000003e-06 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887680+0000", 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5896999999999996e-05 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.887704+0000", 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3516e-05 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890888+0000", 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031837860000000001 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.890936+0000", 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8464000000000002e-05 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.533271999, 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10260 3.9 e40/21 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head v 40'55, mlcod=38'53)", 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9527220449999998, 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027588550000000002, 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225356+0000", 2024-04-24T00:34:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.225357+0000", 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3319999999999999e-06 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226434+0000", 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010764730000000001 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226460+0000", 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6478e-05 2024-04-24T00:34:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226476+0000", 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5769000000000001e-05 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226662+0000", 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186202 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.226706+0000", 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3760000000000001e-05 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.228025+0000", 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013191699999999999 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.228115+0000", 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9671000000000003e-05 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10336 3.9 e40/21 3:912611d7:::rbd_data.115e380d621c.0000000000000213:head v 40'56, mlcod=39'54)", 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7623702219999995, 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021373809999999998, 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415708+0000", 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415710+0000", 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.973e-06 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415797+0000", 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7194999999999994e-05 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415825+0000", 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7501999999999999e-05 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415838+0000", 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3232e-05 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415929+0000", 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1360000000000006e-05 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.415962+0000", 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2570999999999997e-05 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.417692+0000", 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001729801 2024-04-24T00:34:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.417845+0000", 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153747 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10361 3.1f e40/21 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head v 40'65, mlcod=40'63)", 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7218581680000007, 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020517270000000002, 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456220+0000", 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456223+0000", 2024-04-24T00:34:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6970000000000002e-06 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456269+0000", 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.617e-05 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456302+0000", 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3028000000000003e-05 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456323+0000", 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456451+0000", 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012818899999999999 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456512+0000", 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0680000000000002e-05 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458154+0000", 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016422330000000001 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458272+0000", 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117477 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10425 3.4 e41/21 3:2657577c:::rbd_data.115e380d621c.0000000000000210:head v 41'28, mlcod=39'26)", 2024-04-24T00:34:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.5598013250000005, 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041008809999999998, 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618285+0000", 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5580000000000003e-06 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618309+0000", 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-05 2024-04-24T00:34:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618321+0000", 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.164e-05 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619328+0000", 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010071259999999999 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619360+0000", 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1581000000000001e-05 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622325+0000", 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029648610000000001 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622378+0000", 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3235000000000001e-05 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10415 3.1f e41/21 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head v 41'66, mlcod=40'64)", 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.5585626089999991, 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029607800000000001, 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619516+0000", 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.53e-07 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619520+0000", 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3710000000000002e-06 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619530+0000", 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4609999999999997e-06 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619536+0000", 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2879999999999998e-06 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619618+0000", 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2091999999999999e-05 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619640+0000", 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2042e-05 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622396+0000", 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027557300000000001 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622476+0000", 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0543000000000005e-05 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10531 3.9 e41/21 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head v 41'60, mlcod=41'58)", 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.398418135, 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030187019999999998, 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779660+0000", 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779662+0000", 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6819999999999999e-06 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779672+0000", 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0482999999999999e-05 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779700+0000", 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8218999999999999e-05 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779713+0000", 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2123000000000001e-05 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779827+0000", 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114355 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.779866+0000", 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9449000000000001e-05 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.782618+0000", 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027520980000000001 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.782679+0000", 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0293e-05 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10541 3.1f e41/21 3:f85c3e31:::rbd_data.115e380d621c.0000000000000141:head v 41'69, mlcod=41'67)", 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.3926006260000001, 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025351169999999999, 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785478+0000", 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785480+0000", 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2879999999999999e-06 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785493+0000", 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3036e-05 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785529+0000", 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6043e-05 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785546+0000", 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6897000000000001e-05 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785630+0000", 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4124999999999998e-05 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.785657+0000", 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.724e-05 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.787959+0000", 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023013920000000002 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.788013+0000", 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4095999999999998e-05 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10584 3.1f e41/21 3:f85c3e31:::rbd_data.115e380d621c.0000000000000141:head v 41'71, mlcod=41'69)", 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.848796+0000", 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.3292820059999997, 2024-04-24T00:34:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019548859999999999, 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848796+0000", 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848796+0000", 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848797+0000", 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.212e-06 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848812+0000", 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4834999999999999e-05 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848829+0000", 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6580999999999999e-05 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848837+0000", 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0530000000000007e-06 2024-04-24T00:34:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.848962+0000", 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012488399999999999 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.849033+0000", 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0970000000000007e-05 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.850689+0000", 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001656343 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.850751+0000", 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2008000000000007e-05 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10596 3.1f e41/21 3:ff376758:::rbd_data.115e380d621c.000000000000028b:head v 41'72, mlcod=41'70)", 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.926625+0000", 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.2514532169999999, 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019602, 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926625+0000", 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926625+0000", 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926628+0000", 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8629999999999999e-06 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926683+0000", 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.533e-05 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926715+0000", 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1498000000000002e-05 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926732+0000", 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6869000000000001e-05 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926818+0000", 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6213999999999998e-05 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.926842+0000", 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.425e-05 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.928534+0000", 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016915350000000001 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.928585+0000", 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1641000000000002e-05 2024-04-24T00:34:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10672 3.4 e41/21 3:20eab9c7:::rbd_data.115e380d621c.00000000000002b6:head v 41'29, mlcod=39'27)", 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.087977+0000", 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.0901007570000001, 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019443500000000001, 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.087977+0000", 2024-04-24T00:34:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.087977+0000", 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.087979+0000", 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3990000000000001e-06 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.087992+0000", 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3286e-05 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.088014+0000", 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2022999999999999e-05 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.088024+0000", 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0087e-05 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.088153+0000", 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128882 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.088226+0000", 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2484999999999995e-05 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.089804+0000", 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015788009999999999 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.089922+0000", 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011738700000000001 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10682 3.9 e41/21 3:9048ec18:::rbd_data.115e380d621c.000000000000038f:head v 41'62, mlcod=41'60)", 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.110930+0000", 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.0671478949999997, 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020353319999999999, 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110930+0000", 2024-04-24T00:34:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110930+0000", 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110932+0000", 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.114e-06 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110958+0000", 2024-04-24T00:34:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5707e-05 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110982+0000", 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4009e-05 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.110996+0000", 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3389e-05 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111062+0000", 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6002999999999998e-05 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111086+0000", 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4983000000000001e-05 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.112928+0000", 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001841472 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.112966+0000", 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7654999999999998e-05 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10676 3.4 e41/21 3:20eab9c7:::rbd_data.115e380d621c.00000000000002b6:head v 41'30, mlcod=41'28)", 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.111498+0000", 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.0665806409999998, 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021122799999999998, 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111498+0000", 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111498+0000", 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111498+0000", 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111505+0000", 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949999999999999e-06 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111517+0000", 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1588e-05 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111524+0000", 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4880000000000001e-06 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111614+0000", 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9141000000000006e-05 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.111636+0000", 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2532999999999999e-05 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.113578+0000", 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019419509999999999 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.113610+0000", 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.171e-05 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10705 3.9 e41/21 3:93f8bbf0:::rbd_data.115e380d621c.0000000000000399:head v 41'63, mlcod=41'61)", 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.0158412999999999, 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034786589999999998, 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162238+0000", 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162284+0000", 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6122999999999999e-05 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162296+0000", 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1871e-05 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162304+0000", 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7030000000000006e-06 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162390+0000", 2024-04-24T00:34:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6582999999999994e-05 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162431+0000", 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0577999999999997e-05 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165655+0000", 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003224612 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165716+0000", 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0251999999999998e-05 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10758 3.9 e41/21 3:93c11b29:::rbd_data.115e380d621c.0000000000000042:head v 41'65, mlcod=41'63)", 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.266288+0000", 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.9117903170000003, 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034766659999999998, 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266288+0000", 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266288+0000", 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266290+0000", 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9e-06 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266342+0000", 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1823000000000002e-05 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266377+0000", 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5148000000000003e-05 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266392+0000", 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5325999999999999e-05 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266548+0000", 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015587500000000001 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.266616+0000", 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8516000000000006e-05 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.269634+0000", 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003017572 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.269765+0000", 2024-04-24T00:34:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013050599999999999 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:16475 3.9 e45/21 3:91599d2a:::rbd_data.115e8003c269.0000000000000005:head v 45'73, mlcod=45'71)", 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:18.669533+0000", 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.5085454679999999, 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020205610000000001, 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.669533+0000", 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.669533+0000", 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.669536+0000", 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8729999999999999e-06 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.669555+0000", 2024-04-24T00:34:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9286000000000001e-05 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.669604+0000", 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8980999999999997e-05 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.669626+0000", 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2565000000000002e-05 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.669788+0000", 2024-04-24T00:34:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000161698 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.669849+0000", 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1193999999999997e-05 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.671424+0000", 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001574403 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.671553+0000", 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012956099999999999 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:16522 3.9 e45/21 3:91599d2a:::rbd_data.115e8003c269.0000000000000005:head v 45'74, mlcod=45'72)", 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:18.740158+0000", 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.4379200989999998, 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017970149999999999, 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.740158+0000", 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.740158+0000", 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.740160+0000", 2024-04-24T00:34:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4270000000000001e-06 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.740169+0000", 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2529999999999993e-06 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.740187+0000", 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8675999999999999e-05 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.740197+0000", 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8970000000000003e-06 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.740290+0000", 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2509999999999993e-05 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.740320+0000", 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9774999999999999e-05 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.741903+0000", 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001583802 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:18.741955+0000", 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1675000000000002e-05 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.262 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.263 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:22.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:22.265 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.265 INFO:teuthology.orchestra.run.smithi064.stdout:916 write 0x2bf431e thru 0x2bfe3f2 (0xa0d5 bytes) 2024-04-24T00:34:22.265 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:22.259+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:22.277 INFO:teuthology.orchestra.run.smithi064.stdout:918 read 0x71feab4 thru 0x7206013 (0x7560 bytes) 2024-04-24T00:34:22.278 INFO:teuthology.orchestra.run.smithi064.stdout:921 write 0x950fbe1 thru 0x951b60d (0xba2d bytes) 2024-04-24T00:34:22.286 INFO:teuthology.orchestra.run.smithi064.stdout:922 punch from 0x5f8e6c4 to 0x5f9537f, (0x6cbb bytes) 2024-04-24T00:34:22.286 INFO:teuthology.orchestra.run.smithi064.stdout:923 punch from 0xad6505c to 0xad6b18a, (0x612e bytes) 2024-04-24T00:34:22.286 INFO:teuthology.orchestra.run.smithi064.stdout:924 punch from 0x1c1c987 to 0x1c249a0, (0x8019 bytes) 2024-04-24T00:34:22.287 INFO:teuthology.orchestra.run.smithi064.stdout:926 writesame 0x1a6a717 thru 0x1a7293e data_size 0x8c(0x8228 bytes) 2024-04-24T00:34:22.290 INFO:teuthology.orchestra.run.smithi064.stdout:927 punch from 0x4f63a50 to 0x4f6e87a, (0xae2a bytes) 2024-04-24T00:34:22.290 INFO:teuthology.orchestra.run.smithi064.stdout:928 write 0xda6a055 thru 0xda6a123 (0xcf bytes) 2024-04-24T00:34:22.295 INFO:teuthology.orchestra.run.smithi064.stdout:931 read 0x40687a2 thru 0x4070dc2 (0x8621 bytes) 2024-04-24T00:34:22.296 INFO:teuthology.orchestra.run.smithi064.stdout:932 write 0x42da25 thru 0x42da5d (0x39 bytes) 2024-04-24T00:34:22.299 INFO:teuthology.orchestra.run.smithi064.stdout:934 punch from 0xc13ba04 to 0xc148179, (0xc775 bytes) 2024-04-24T00:34:22.299 INFO:teuthology.orchestra.run.smithi064.stdout:936 write 0x4895bc4 thru 0x4899be4 (0x4021 bytes) 2024-04-24T00:34:22.314 INFO:teuthology.orchestra.run.smithi064.stdout:937 trunc from 0xdddf5ae to 0x1bd9162 2024-04-24T00:34:22.361 INFO:teuthology.orchestra.run.smithi064.stdout:940 trunc from 0x1bd9162 to 0x5abd233 2024-04-24T00:34:22.364 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:22.360+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:22.388 INFO:teuthology.orchestra.run.smithi064.stdout:941 writesame 0x8db8a2d thru 0x8dc6c7d data_size 0x17(0xe251 bytes) 2024-04-24T00:34:22.388 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.392 INFO:teuthology.orchestra.run.smithi064.stdout:946 trunc from 0x8dc6c7e to 0xca0f614 2024-04-24T00:34:22.405 INFO:teuthology.orchestra.run.smithi064.stdout:947 read 0x5c9f873 thru 0x5cabb7f (0xc30d bytes) 2024-04-24T00:34:22.405 INFO:teuthology.orchestra.run.smithi064.stdout:948 read 0x4428b1f thru 0x442f02b (0x650d bytes) 2024-04-24T00:34:22.408 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:22.408 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:22.408 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:22.408 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:22.420 INFO:teuthology.orchestra.run.smithi064.stdout:949 trunc from 0xca0f614 to 0x21c5132 2024-04-24T00:34:22.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:22.458 INFO:teuthology.orchestra.run.smithi064.stdout:950 write 0x5aedb01 thru 0x5af4da9 (0x72a9 bytes) 2024-04-24T00:34:22.465 INFO:teuthology.orchestra.run.smithi064.stdout:951 read 0x101907 thru 0x10da96 (0xc190 bytes) 2024-04-24T00:34:22.465 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:22.463+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:22.467 INFO:teuthology.orchestra.run.smithi064.stdout:952 read 0x4cf5dd6 thru 0x4d00f44 (0xb16f bytes) 2024-04-24T00:34:22.467 INFO:teuthology.orchestra.run.smithi064.stdout:953 read 0x4ea99da thru 0x4eb4a62 (0xb089 bytes) 2024-04-24T00:34:22.491 INFO:teuthology.orchestra.run.smithi064.stdout:954 writesame 0xab66952 thru 0xab6bb19 data_size 0x4(0x51c8 bytes) 2024-04-24T00:34:22.491 INFO:teuthology.orchestra.run.smithi064.stdout:955 read 0x6101e2c thru 0x610e12d (0xc302 bytes) 2024-04-24T00:34:22.492 INFO:teuthology.orchestra.run.smithi064.stdout:956 read 0x4a379cf thru 0x4a4326d (0xb89f bytes) 2024-04-24T00:34:22.492 INFO:teuthology.orchestra.run.smithi064.stdout:957 punch from 0x8102b08 to 0x810e1a3, (0xb69b bytes) 2024-04-24T00:34:22.493 INFO:teuthology.orchestra.run.smithi064.stdout:958 writesame 0x9c2823b thru 0x9c30d83 data_size 0x1(0x8b49 bytes) 2024-04-24T00:34:22.566 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:22.560+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:22.582 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:22.582 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:22.583 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:22.583 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:22.583 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:22.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:960 punch from 0x1cf84f1 to 0x1d0230f, (0x9e1e bytes) 2024-04-24T00:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:964 writesame 0x244d7d8 thru 0x244e445 data_size 0x1(0xc6e bytes) 2024-04-24T00:34:22.602 INFO:teuthology.orchestra.run.smithi064.stdout:965 write 0x74bffd9 thru 0x74ccf74 (0xcf9c bytes) 2024-04-24T00:34:22.621 INFO:teuthology.orchestra.run.smithi064.stdout:966 trunc from 0xab6bb1a to 0x48c5e94 2024-04-24T00:34:22.660 INFO:teuthology.orchestra.run.smithi064.stdout:967 write 0x8ab2209 thru 0x8ab6b73 (0x496b bytes) 2024-04-24T00:34:22.667 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.667 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:22.663+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:22.673 INFO:teuthology.orchestra.run.smithi064.stdout:969 trunc from 0x8ab6b74 to 0xd41e024 2024-04-24T00:34:22.685 INFO:teuthology.orchestra.run.smithi064.stdout:970 write 0x869be10 thru 0x86a7f74 (0xc165 bytes) 2024-04-24T00:34:22.686 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:34:22.702 INFO:teuthology.orchestra.run.smithi064.stdout:971 write 0x679e1f thru 0x687487 (0xd669 bytes) 2024-04-24T00:34:22.716 INFO:teuthology.orchestra.run.smithi064.stdout:972 trunc from 0xd41e024 to 0x6d8227f 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2036 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.206382938000001, 2024-04-24T00:34:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020639231000000001, 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2036 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528464+0000", 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600000000000004e-07 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528467+0000", 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8040000000000002e-06 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528474+0000", 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7270000000000007e-06 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528480+0000", 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3340000000000001e-06 2024-04-24T00:34:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548902+0000", 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020421963000000001 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548991+0000", 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9332000000000005e-05 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549102+0000", 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111535 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2053 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.206122767, 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616257999999998, 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2053 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528724+0000", 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6400000000000002e-07 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528727+0000", 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6749999999999998e-06 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528734+0000", 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4739999999999999e-06 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528740+0000", 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4759999999999997e-06 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549138+0000", 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020398315 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549225+0000", 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6802e-05 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549340+0000", 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011495200000000001 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2054 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.206098901000001, 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020836107, 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2054 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528748+0000", 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999999e-07 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528750+0000", 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528756+0000", 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1249999999999998e-06 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528761+0000", 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549377+0000", 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616254000000001 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549468+0000", 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1199000000000006e-05 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549583+0000", 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011484099999999999 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2055 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.206077920999999, 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022112625, 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2055 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528769+0000", 2024-04-24T00:34:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9299999999999999e-07 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528771+0000", 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2809999999999998e-06 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528777+0000", 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.127e-06 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528782+0000", 2024-04-24T00:34:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8330000000000001e-06 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550648+0000", 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021866450999999999 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550755+0000", 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010676800000000001 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550881+0000", 2024-04-24T00:34:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012577199999999999 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2056 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.206056202999999, 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022390911999999999, 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2056 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528792+0000", 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1390000000000002e-06 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528798+0000", 2024-04-24T00:34:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9869999999999996e-06 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528804+0000", 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4949999999999999e-06 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550928+0000", 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022124061 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551050+0000", 2024-04-24T00:34:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121981 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551181+0000", 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130858 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.206035258, 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:34:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:34:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.206011344, 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:34:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:34:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205984170000001, 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205958065000001, 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205547083999999, 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205527275000001, 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205474215999999, 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:34:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:34:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.20543936, 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:34:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:34:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:34:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205417795999999, 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:34:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205388304, 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:34:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205367442, 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.205344868000001, 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:34:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5230853700000004, 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:34:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:34:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:34:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:34:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5227344550000002, 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:34:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:34:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5158461350000003, 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:22.825 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:22.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:22.827 INFO:teuthology.orchestra.run.smithi064.stdout:973 write 0xba829f9 thru 0xba90a16 (0xe01e bytes) 2024-04-24T00:34:22.827 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.827 INFO:teuthology.orchestra.run.smithi064.stdout:975 punch from 0x161210f to 0x161d5d4, (0xb4c5 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:978 read 0x73cfd83 thru 0x73dc744 (0xc9c2 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:979 read 0x5f752f2 thru 0x5f837a7 (0xe4b6 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:980 trunc from 0xba90a17 to 0x59c4d3a 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:981 write 0xb75954 thru 0xb7f29c (0x9949 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:982 read 0x27a3579 thru 0x27afca9 (0xc731 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:984 read 0x256b5e8 thru 0x2574baf (0x95c8 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:985 read 0xfd193 thru 0x100ec4 (0x3d32 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:986 read 0x46dad32 thru 0x46e97e3 (0xeab2 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:987 write 0x608f5b3 thru 0x6092c77 (0x36c5 bytes) 2024-04-24T00:34:22.828 INFO:teuthology.orchestra.run.smithi064.stdout:989 write 0x11ca53d thru 0x11cb2da (0xd9e bytes) 2024-04-24T00:34:22.828 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:22.823+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:22.829 INFO:teuthology.orchestra.run.smithi064.stdout:990 punch from 0x28384c5 to 0x283a3c3, (0x1efe bytes) 2024-04-24T00:34:22.843 INFO:teuthology.orchestra.run.smithi064.stdout:991 write 0x81a9778 thru 0x81ada4f (0x42d8 bytes) 2024-04-24T00:34:22.848 INFO:teuthology.orchestra.run.smithi064.stdout:992 punch from 0x25ffed0 to 0x26074a0, (0x75d0 bytes) 2024-04-24T00:34:22.857 INFO:teuthology.orchestra.run.smithi064.stdout:993 trunc from 0x81ada50 to 0xd3f1aaf 2024-04-24T00:34:22.868 INFO:teuthology.orchestra.run.smithi064.stdout:994 write 0x5423a18 thru 0x542f48f (0xba78 bytes) 2024-04-24T00:34:22.875 INFO:teuthology.orchestra.run.smithi064.stdout:995 write 0xac81741 thru 0xac88c40 (0x7500 bytes) 2024-04-24T00:34:22.883 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.883 INFO:teuthology.orchestra.run.smithi064.stdout:997 read 0x339180b thru 0x339c5ac (0xada2 bytes) 2024-04-24T00:34:22.883 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.884 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.884 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.896 INFO:teuthology.orchestra.run.smithi064.stdout:1002 trunc from 0xd3f1aaf to 0x71652e3 2024-04-24T00:34:22.916 INFO:teuthology.orchestra.run.smithi064.stdout:1004 read 0x4da2994 thru 0x4db0516 (0xdb83 bytes) 2024-04-24T00:34:22.916 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.916 INFO:teuthology.orchestra.run.smithi064.stdout:1006 read 0x3833da6 thru 0x384092b (0xcb86 bytes) 2024-04-24T00:34:22.917 INFO:teuthology.orchestra.run.smithi064.stdout:1008 punch from 0x5fdd6d1 to 0x5fe7bba, (0xa4e9 bytes) 2024-04-24T00:34:22.920 INFO:teuthology.orchestra.run.smithi064.stdout:1009 trunc from 0x71652e3 to 0x8a1485b 2024-04-24T00:34:22.926 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:22.920+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:22.944 INFO:teuthology.orchestra.run.smithi064.stdout:1010 trunc from 0x8a1485b to 0x287724b 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20544 3.1c 3:3b4487a7:::rbd_object_map.115ee4742374:head [call lock.assert_locked in=44b,call rbd.object_map_resize in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:22.965260+0000", 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00191866, 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019384929999999999, 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:22.969 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20544 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965260+0000", 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965260+0000", 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965262+0000", 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4759999999999999e-06 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965267+0000", 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6820000000000002e-06 2024-04-24T00:34:22.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965279+0000", 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2101e-05 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965284+0000", 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7180000000000001e-06 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965353+0000", 2024-04-24T00:34:22.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9029000000000006e-05 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965457+0000", 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010354 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.965737+0000", 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028015700000000001 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:22.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:22.966743+0000", 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001005431 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:34:22.974 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:22.975 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:22.975 INFO:teuthology.orchestra.run.smithi064.stdout:1013 write 0x2106602 thru 0x21136e1 (0xd0e0 bytes) 2024-04-24T00:34:22.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:23.006 INFO:teuthology.orchestra.run.smithi064.stdout:1015 trunc from 0x287724b to 0xd111712 2024-04-24T00:34:23.018 INFO:teuthology.orchestra.run.smithi064.stdout:1016 read 0x452f396 thru 0x45398d9 (0xa544 bytes) 2024-04-24T00:34:23.018 INFO:teuthology.orchestra.run.smithi064.stdout:1017 read 0xc065816 thru 0xc06b2e6 (0x5ad1 bytes) 2024-04-24T00:34:23.018 INFO:teuthology.orchestra.run.smithi064.stdout:1018 writesame 0x2e4aa35 thru 0x2e546f5 data_size 0x1(0x9cc1 bytes) 2024-04-24T00:34:23.027 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:23.023+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:23.125 INFO:teuthology.orchestra.run.smithi064.stdout:1022 trunc from 0xd111712 to 0x3a7de6f 2024-04-24T00:34:23.128 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:23.124+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:23.142 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:23.166 INFO:teuthology.orchestra.run.smithi064.stdout:1026 read 0x1b9c32b thru 0x1ba17ac (0x5482 bytes) 2024-04-24T00:34:23.167 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:23.167 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:23.167 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:23.167 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:23.167 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:23.167 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:23.171 INFO:teuthology.orchestra.run.smithi064.stdout:1028 writesame 0x8ec2633 thru 0x8ec2c6f data_size 0x1(0x63d bytes) 2024-04-24T00:34:23.171 INFO:teuthology.orchestra.run.smithi064.stdout:1030 read 0x4444b65 thru 0x4450230 (0xb6cc bytes) 2024-04-24T00:34:23.171 INFO:teuthology.orchestra.run.smithi064.stdout:1031 write 0x2d1ae05 thru 0x2d21472 (0x666e bytes) 2024-04-24T00:34:23.172 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:23.174 INFO:teuthology.orchestra.run.smithi064.stdout:1032 read 0xb06671 thru 0xb0d5f4 (0x6f84 bytes) 2024-04-24T00:34:23.176 INFO:teuthology.orchestra.run.smithi064.stdout:1034 read 0x849c2e0 thru 0x84a319a (0x6ebb bytes) 2024-04-24T00:34:23.188 INFO:teuthology.orchestra.run.smithi064.stdout:1036 write 0x99fc8d5 thru 0x9a02edd (0x6609 bytes) 2024-04-24T00:34:23.217 INFO:teuthology.orchestra.run.smithi064.stdout:1037 write 0xd113856 thru 0xd114f52 (0x16fd bytes) 2024-04-24T00:34:23.224 INFO:teuthology.orchestra.run.smithi064.stdout:1038 punch from 0x8cb9346 to 0x8cc23d0, (0x908a bytes) 2024-04-24T00:34:23.224 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:23.224 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:23.224 INFO:teuthology.orchestra.run.smithi064.stdout:1042 writesame 0x56a1ad3 thru 0x56a6914 data_size 0x6a(0x4e42 bytes) 2024-04-24T00:34:23.230 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:23.227+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:23.232 INFO:teuthology.orchestra.run.smithi064.stdout:1043 write 0x5ac5d76 thru 0x5acd40d (0x7698 bytes) 2024-04-24T00:34:23.241 INFO:teuthology.orchestra.run.smithi064.stdout:1045 punch from 0xb73569f to 0xb742425, (0xcd86 bytes) 2024-04-24T00:34:23.274 INFO:teuthology.orchestra.run.smithi064.stdout:1048 trunc from 0xd114f53 to 0x3cf9f9 2024-04-24T00:34:23.317 INFO:teuthology.orchestra.run.smithi064.stdout:1049 trunc from 0x3cf9f9 to 0x68961fe 2024-04-24T00:34:23.331 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:23.328+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:23.344 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:23.344 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3338 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13434880~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.511063754999999, 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021864103999999999, 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3338 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3699999999999999e-07 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828608+0000", 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.216e-06 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828619+0000", 2024-04-24T00:34:23.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.172e-05 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828627+0000", 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.79e-06 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850192+0000", 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021564967000000001 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850314+0000", 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121789 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850467+0000", 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015268500000000001 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3339 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13565952~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:23.349 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.511028092, 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022319400999999999, 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3339 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828639+0000", 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0100000000000001e-06 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828643+0000", 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2300000000000002e-06 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828652+0000", 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000001e-06 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828659+0000", 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850539+0000", 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021879770999999999 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850647+0000", 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010767 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850958+0000", 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031074100000000001 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3400 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13828096~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:23.353 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.504231672, 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022009407000000002, 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3400 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835436+0000", 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8139999999999999e-06 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835462+0000", 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5273e-05 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835489+0000", 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6859000000000002e-05 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835507+0000", 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8144000000000001e-05 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.856821+0000", 2024-04-24T00:34:23.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021313959 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857070+0000", 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024914099999999998 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857444+0000", 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037421699999999998 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3401 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13959168~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.504135196, 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022623199, 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3401 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835532+0000", 2024-04-24T00:34:23.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3769999999999999e-06 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835540+0000", 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2969999999999998e-06 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835559+0000", 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9199999999999999e-05 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835574+0000", 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4983e-05 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857648+0000", 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022074522999999999 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857816+0000", 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016745200000000001 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858154+0000", 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033836700000000001 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3402 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14090240~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.504072336, 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023133308000000002, 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.362 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3402 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835595+0000", 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.043e-06 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835602+0000", 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7329999999999997e-06 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835624+0000", 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2079999999999999e-05 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835638+0000", 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4270000000000001e-05 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858219+0000", 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022580586 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858442+0000", 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000223191 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858727+0000", 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028540500000000002 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3403 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14221312~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.50400892, 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023734209999999999, 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.366 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3403 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835658+0000", 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.083e-06 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835664+0000", 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5999999999999997e-06 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835683+0000", 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8604e-05 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835697+0000", 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4576000000000001e-05 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858838+0000", 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023140524999999999 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859063+0000", 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000225598 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859392+0000", 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032822400000000001 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3404 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14352384~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.503950370999998, 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024301263, 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.370 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3404 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835717+0000", 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835760+0000", 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2993e-05 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835781+0000", 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0743000000000001e-05 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835796+0000", 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5233e-05 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859493+0000", 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023697296999999999 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859722+0000", 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000228619 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860017+0000", 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029529399999999998 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3405 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14483456~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.503847020999999, 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024591302999999998, 2024-04-24T00:34:23.374 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3405 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835821+0000", 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835827+0000", 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5300000000000002e-06 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835845+0000", 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8096000000000001e-05 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835859+0000", 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3916e-05 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860117+0000", 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024257616999999999 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860241+0000", 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012400200000000001 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860411+0000", 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169562 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3406 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14614528~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:23.378 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.503786712, 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024976411, 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3406 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835881+0000", 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835887+0000", 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3369999999999996e-06 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835905+0000", 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7592999999999999e-05 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835919+0000", 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4277000000000001e-05 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860500+0000", 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024580722999999999 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860651+0000", 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015075900000000001 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860856+0000", 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020540200000000001 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3455 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.500967190000001, 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022775208000000002, 2024-04-24T00:34:23.382 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3455 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838702+0000", 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838715+0000", 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2959000000000001e-05 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838740+0000", 2024-04-24T00:34:23.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4893e-05 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838753+0000", 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3176999999999999e-05 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860986+0000", 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022233541999999999 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.861203+0000", 2024-04-24T00:34:23.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000216472 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.861474+0000", 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027145799999999997 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.698928234, 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:34:23.386 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:34:23.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:34:23.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.698852035, 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:34:23.390 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.698393861, 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:34:23.394 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:34:23.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:23.398 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.698336554000001, 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:23.402 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.698077588, 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.697156116, 2024-04-24T00:34:23.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.125341386000001, 2024-04-24T00:34:23.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:34:23.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:34:23.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:23.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.109812137, 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.416 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:34:23.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.4810888169999998, 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:34:23.420 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:34:23.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:34:23.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:23.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.4660536900000007, 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:34:23.426 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:23.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:34:23.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:34:23.430 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.430 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.430 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.430 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:23.430 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:23.430 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:23.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:23.432 INFO:teuthology.orchestra.run.smithi064.stdout:1050 trunc from 0x68961fe to 0xd7ffede 2024-04-24T00:34:23.432 INFO:teuthology.orchestra.run.smithi064.stdout:1051 writesame 0x9f92907 thru 0x9f942df data_size 0x1(0x19d9 bytes) 2024-04-24T00:34:23.432 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:23.432 INFO:teuthology.orchestra.run.smithi064.stdout:1054 write 0x1f63eaa thru 0x1f6d65c (0x97b3 bytes) 2024-04-24T00:34:23.432 INFO:teuthology.orchestra.run.smithi064.stdout:1055 trunc from 0xd7ffede to 0x7f26aed 2024-04-24T00:34:23.432 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:23.432 INFO:teuthology.orchestra.run.smithi064.stdout:1059 read 0x6f82ec9 thru 0x6f9221b (0xf353 bytes) 2024-04-24T00:34:23.432 INFO:teuthology.orchestra.run.smithi064.stdout:1060 punch from 0x7c98f4d to 0x7ca1133, (0x81e6 bytes) 2024-04-24T00:34:23.433 INFO:teuthology.orchestra.run.smithi064.stdout:1061 read 0x6012ed1 thru 0x6019c0d (0x6d3d bytes) 2024-04-24T00:34:23.433 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:23.433 INFO:teuthology.orchestra.run.smithi064.stdout:1065 clone 7 order 22 su 1048576 sc 4 2024-04-24T00:34:23.475 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:23.472+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:23.534 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:23.528+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:23.601 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:23.602 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:23.602 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:23.602 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:23.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:23.635 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:23.631+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:23.736 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:23.732+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:23.790 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:23.790 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:23.790 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:23.790 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:23.790 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:23.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:23.838 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:23.835+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:23.939 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:23.935+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:23.969 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:23.969 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:23.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:23.969 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [] 2024-04-24T00:34:23.969 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:23.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:24.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:24.039 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:24.036+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:24.110 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:24.111 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:24.111 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:24.125 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:24.140 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:24.135+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:24.168 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:24.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:24.186 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:24.186 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:24.204 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:24.204 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:24.221 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:24.221 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:24.240 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:24.236+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:24.241 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:24.241 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:24.259 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:24.259 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:24.280 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:24.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:24.295 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:24.295 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:24.301 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:24.302 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:24.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:24.351 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:24.348+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:24.442 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:24.436+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:24.473 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:24.473 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:24.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:24.575 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:24.572+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:24.643 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:24.640+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:24.675 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:24.676 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:24.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:24.707 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:34:24.753 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:24.751+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:24.845 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:24.839+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:24.858 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:24.858 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:24.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:24.945 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:24.943+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:24.995 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:24.995 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:25.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:25.046 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:25.040+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:25.140 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:25.141 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:25.147 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:25.143+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:25.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:25.248 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:25.244+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:25.324 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:25.324 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:25.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:25.349 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:25.344+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:25.450 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:25.444+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:25.465 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:25.465 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:25.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:25.539 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:34:25.550 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:25.547+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:25.620 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:25.621 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:25.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:25.681 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:25.679+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:25.684 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-04-24T00:34:25.751 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:25.748+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:25.761 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:25.761 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:25.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:25.852 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:25.848+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:25.943 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:25.944 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:25.953 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:25.951+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:25.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:26.054 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:26.052+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:26.116 INFO:teuthology.orchestra.run.smithi064.stdout:1066 trunc from 0x7f26aed to 0xc1cd301 2024-04-24T00:34:26.129 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:26.129 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:26.137 INFO:teuthology.orchestra.run.smithi064.stdout:1067 punch from 0x10c20a3 to 0x10cca0c, (0xa969 bytes) 2024-04-24T00:34:26.138 INFO:teuthology.orchestra.run.smithi064.stdout:1068 punch from 0x50aa2d1 to 0x50b3c40, (0x996f bytes) 2024-04-24T00:34:26.138 INFO:teuthology.orchestra.run.smithi064.stdout:1069 writesame 0x292cf53 thru 0x293a357 data_size 0x1(0xd405 bytes) 2024-04-24T00:34:26.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:26.193 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:26.191+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:26.257 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:26.252+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:26.283 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:26.283 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:26.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:26.316 INFO:teuthology.orchestra.run.smithi064.stdout:1070 read 0xbbc347 thru 0xbc3efe (0x7bb8 bytes) 2024-04-24T00:34:26.317 INFO:teuthology.orchestra.run.smithi064.stdout:1071 punch from 0x3f40f7f to 0x3f43049, (0x20ca bytes) 2024-04-24T00:34:26.317 INFO:teuthology.orchestra.run.smithi064.stdout:1072 write 0xbd68406 thru 0xbd6f2d2 (0x6ecd bytes) 2024-04-24T00:34:26.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:26.327 INFO:teuthology.orchestra.run.smithi064.stdout:1073 punch from 0x21a38d5 to 0x21a7e3c, (0x4567 bytes) 2024-04-24T00:34:26.342 INFO:teuthology.orchestra.run.smithi064.stdout:1078 writesame 0xcc999ec thru 0xcca7725 data_size 0x3(0xdd3a bytes) 2024-04-24T00:34:26.342 INFO:teuthology.orchestra.run.smithi064.stdout:1081 read 0x7321b16 thru 0x732cb72 (0xb05d bytes) 2024-04-24T00:34:26.357 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:26.352+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:26.359 INFO:teuthology.orchestra.run.smithi064.stdout:1082 write 0xe9ca1b8 thru 0xe9d9b36 (0xf97f bytes) 2024-04-24T00:34:26.367 INFO:teuthology.orchestra.run.smithi064.stdout:1084 read 0x21786e4 thru 0x2178f7e (0x89b bytes) 2024-04-24T00:34:26.367 INFO:teuthology.orchestra.run.smithi064.stdout:1085 writesame 0xbbfe1bc thru 0xbc05343 data_size 0x38(0x7188 bytes) 2024-04-24T00:34:26.375 INFO:teuthology.orchestra.run.smithi064.stdout:1086 write 0xda73f31 thru 0xda7a34c (0x641c bytes) 2024-04-24T00:34:26.382 INFO:teuthology.orchestra.run.smithi064.stdout:1088 writesame 0xf425c4 thru 0xf4d716 data_size 0x7(0xb153 bytes) 2024-04-24T00:34:26.400 INFO:teuthology.orchestra.run.smithi064.stdout:1091 write 0x7565544 thru 0x75718b5 (0xc372 bytes) 2024-04-24T00:34:26.412 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:26.412 INFO:teuthology.orchestra.run.smithi064.stdout:1096 write 0xd26a536 thru 0xd270b09 (0x65d4 bytes) 2024-04-24T00:34:26.415 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:26.417 INFO:teuthology.orchestra.run.smithi064.stdout:1099 trunc from 0xe9d9b37 to 0xd45763e 2024-04-24T00:34:26.444 INFO:teuthology.orchestra.run.smithi064.stdout:1100 writesame 0x4e60317 thru 0x4e64092 data_size 0xa(0x3d7c bytes) 2024-04-24T00:34:26.447 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:26.447 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:26.457 INFO:teuthology.orchestra.run.smithi064.stdout:1101 writesame 0x8061199 thru 0x806a854 data_size 0x16(0x96bc bytes) 2024-04-24T00:34:26.457 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:26.452+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:26.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:26.464 INFO:teuthology.orchestra.run.smithi064.stdout:1102 write 0x8aa3127 thru 0x8aaa0ee (0x6fc8 bytes) 2024-04-24T00:34:26.471 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:26.471 INFO:teuthology.orchestra.run.smithi064.stdout:1105 punch from 0x9328490 to 0x9332ec8, (0xaa38 bytes) 2024-04-24T00:34:26.471 INFO:teuthology.orchestra.run.smithi064.stdout:1106 punch from 0x16982f0 to 0x169ef05, (0x6c15 bytes) 2024-04-24T00:34:26.472 INFO:teuthology.orchestra.run.smithi064.stdout:1107 punch from 0x5bb6205 to 0x5bc5659, (0xf454 bytes) 2024-04-24T00:34:26.472 INFO:teuthology.orchestra.run.smithi064.stdout:1108 trunc from 0xd45763e to 0xe00e21d 2024-04-24T00:34:26.473 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:26.485 INFO:teuthology.orchestra.run.smithi064.stdout:1109 read 0x7bba698 thru 0x7bbd147 (0x2ab0 bytes) 2024-04-24T00:34:26.492 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:26.492 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:26.496 INFO:teuthology.orchestra.run.smithi064.stdout:1111 trunc from 0xe00e21d to 0x7e8bfa8 2024-04-24T00:34:26.511 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:26.511 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:26.528 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:26.529 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:26.540 INFO:teuthology.orchestra.run.smithi064.stdout:1112 write 0x887100a thru 0x887c086 (0xb07d bytes) 2024-04-24T00:34:26.547 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:26.547 INFO:teuthology.orchestra.run.smithi064.stdout:1115 read 0x700ae27 thru 0x700fe5a (0x5034 bytes) 2024-04-24T00:34:26.548 INFO:teuthology.orchestra.run.smithi064.stdout:1116 read 0x5190836 thru 0x519308f (0x285a bytes) 2024-04-24T00:34:26.550 INFO:teuthology.orchestra.run.smithi064.stdout:1117 trunc from 0x887c087 to 0x9b6abc6 2024-04-24T00:34:26.553 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:26.553 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:26.558 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:26.552+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:26.564 INFO:teuthology.orchestra.run.smithi064.stdout:1118 punch from 0x822fc9c to 0x823fb7a, (0xfede bytes) 2024-04-24T00:34:26.564 INFO:teuthology.orchestra.run.smithi064.stdout:1119 read 0x349f99a thru 0x34a4848 (0x4eaf bytes) 2024-04-24T00:34:26.574 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:26.579 INFO:teuthology.orchestra.run.smithi064.stdout:1120 trunc from 0x9b6abc6 to 0x5ef3fe 2024-04-24T00:34:26.599 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:26.611 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:26.612 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:26.613 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:26.620 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:26.632 INFO:teuthology.orchestra.run.smithi064.stdout:1124 trunc from 0x5ef3fe to 0xc144fc2 2024-04-24T00:34:26.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:26.647 INFO:teuthology.orchestra.run.smithi064.stdout:1125 trunc from 0xc144fc2 to 0xcf4be1c 2024-04-24T00:34:26.659 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:26.660 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:26.655+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:26.676 INFO:teuthology.orchestra.run.smithi064.stdout:1127 write 0xebcea71 thru 0xebd5883 (0x6e13 bytes) 2024-04-24T00:34:26.682 INFO:teuthology.orchestra.run.smithi064.stdout:1128 punch from 0xa3644f7 to 0xa372f06, (0xea0f bytes) 2024-04-24T00:34:26.682 INFO:teuthology.orchestra.run.smithi064.stdout:1129 read 0xe0bfa6c thru 0xe0cd853 (0xdde8 bytes) 2024-04-24T00:34:26.683 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:26.683 INFO:teuthology.orchestra.run.smithi064.stdout:1132 read 0x86a425d thru 0x86b2b06 (0xe8aa bytes) 2024-04-24T00:34:26.683 INFO:teuthology.orchestra.run.smithi064.stdout:1133 write 0x84ede50 thru 0x84ede8b (0x3c bytes) 2024-04-24T00:34:26.703 INFO:teuthology.orchestra.run.smithi064.stdout:1134 trunc from 0xebd5884 to 0x750ca23 2024-04-24T00:34:26.725 INFO:teuthology.orchestra.run.smithi064.stdout:1135 read 0x4fdd583 thru 0x4fe9593 (0xc011 bytes) 2024-04-24T00:34:26.725 INFO:teuthology.orchestra.run.smithi064.stdout:1136 writesame 0x637af10 thru 0x6388fcc data_size 0x1(0xe0bd bytes) 2024-04-24T00:34:26.761 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:26.755+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:26.797 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:26.797 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:26.797 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:26.797 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:26.797 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:26.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:26.870 INFO:teuthology.orchestra.run.smithi064.stdout:1137 trunc from 0x750ca23 to 0x7a59cf6 2024-04-24T00:34:26.870 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:26.867+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:26.878 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:26.879 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:26.879 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-24T00:34:26.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-04-24T00:34:26.882 INFO:teuthology.orchestra.run.smithi064.stdout:1139 write 0x4ab5ee9 thru 0x4abb2ec (0x5404 bytes) 2024-04-24T00:34:26.890 INFO:teuthology.orchestra.run.smithi064.stdout:1140 punch from 0x3ca209c to 0x3ca7970, (0x58d4 bytes) 2024-04-24T00:34:26.908 INFO:teuthology.orchestra.run.smithi064.stdout:1143 write 0xb9a6295 thru 0xb9a827f (0x1feb bytes) 2024-04-24T00:34:26.915 INFO:teuthology.orchestra.run.smithi064.stdout:1144 read 0x15def70 thru 0x15e5164 (0x61f5 bytes) 2024-04-24T00:34:26.915 INFO:teuthology.orchestra.run.smithi064.stdout:1145 punch from 0x60ac251 to 0x60af0d9, (0x2e88 bytes) 2024-04-24T00:34:26.915 INFO:teuthology.orchestra.run.smithi064.stdout:1146 write 0x2b7fdba thru 0x2b8e8a1 (0xeae8 bytes) 2024-04-24T00:34:26.930 INFO:teuthology.orchestra.run.smithi064.stdout:1147 write 0x40c3254 thru 0x40c5b25 (0x28d2 bytes) 2024-04-24T00:34:26.937 INFO:teuthology.orchestra.run.smithi064.stdout:1148 writesame 0x3bb3439 thru 0x3bc05a7 data_size 0x1(0xd16f bytes) 2024-04-24T00:34:26.963 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:26.960+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:26.984 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2036 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.45185249, 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020639231000000001, 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2036 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528464+0000", 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600000000000004e-07 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528467+0000", 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8040000000000002e-06 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528474+0000", 2024-04-24T00:34:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7270000000000007e-06 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528480+0000", 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3340000000000001e-06 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548902+0000", 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020421963000000001 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548991+0000", 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9332000000000005e-05 2024-04-24T00:34:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549102+0000", 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111535 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2053 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451592319, 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616257999999998, 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2053 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528724+0000", 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6400000000000002e-07 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528727+0000", 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6749999999999998e-06 2024-04-24T00:34:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528734+0000", 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4739999999999999e-06 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528740+0000", 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4759999999999997e-06 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549138+0000", 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020398315 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549225+0000", 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6802e-05 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549340+0000", 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011495200000000001 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2054 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451568453, 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020836107, 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2054 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528748+0000", 2024-04-24T00:34:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999999e-07 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528750+0000", 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528756+0000", 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1249999999999998e-06 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528761+0000", 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549377+0000", 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616254000000001 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549468+0000", 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1199000000000006e-05 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549583+0000", 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011484099999999999 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2055 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451547473000002, 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022112625, 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:26.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2055 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:26.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528769+0000", 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9299999999999999e-07 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528771+0000", 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2809999999999998e-06 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528777+0000", 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.127e-06 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528782+0000", 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8330000000000001e-06 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550648+0000", 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021866450999999999 2024-04-24T00:34:27.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550755+0000", 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010676800000000001 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550881+0000", 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012577199999999999 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2056 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451525754999999, 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022390911999999999, 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2056 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528792+0000", 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1390000000000002e-06 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528798+0000", 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9869999999999996e-06 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528804+0000", 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4949999999999999e-06 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550928+0000", 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022124061 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551050+0000", 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121981 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551181+0000", 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130858 2024-04-24T00:34:27.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451504809999999, 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:34:27.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:34:27.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:34:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:34:27.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451480896, 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451453722, 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:27.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:34:27.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:34:27.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451427617, 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:27.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:34:27.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:34:27.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451016635999999, 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:27.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.450996827000001, 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:34:27.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.450943767999998, 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:34:27.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:34:27.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.450908911999999, 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:27.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.450887347999998, 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.450857855999999, 2024-04-24T00:34:27.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.048 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.450836993999999, 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:27.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:34:27.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.45081442, 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:34:27.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:34:27.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.768554922, 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:34:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.768204007, 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:34:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:34:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.761315687, 2024-04-24T00:34:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:34:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:27.073 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:27.074 INFO:teuthology.orchestra.run.smithi064.stdout:1149 read 0xfab0a5 thru 0xfb4416 (0x9372 bytes) 2024-04-24T00:34:27.074 INFO:teuthology.orchestra.run.smithi064.stdout:1151 read 0x11b9a12 thru 0x11bde22 (0x4411 bytes) 2024-04-24T00:34:27.074 INFO:teuthology.orchestra.run.smithi064.stdout:1154 write 0xac33c03 thru 0xac40c24 (0xd022 bytes) 2024-04-24T00:34:27.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:27.076 INFO:teuthology.orchestra.run.smithi064.stdout:1155 writesame 0xecc0e4 thru 0xece63c data_size 0x1(0x2559 bytes) 2024-04-24T00:34:27.076 INFO:teuthology.orchestra.run.smithi064.stdout:1156 read 0x159d296 thru 0x15a4474 (0x71df bytes) 2024-04-24T00:34:27.076 INFO:teuthology.orchestra.run.smithi064.stdout:1157 write 0xddf964b thru 0xde063f5 (0xcdab bytes) 2024-04-24T00:34:27.076 INFO:teuthology.orchestra.run.smithi064.stdout:1158 writesame 0x5214125 thru 0x521e0c7 data_size 0x1(0x9fa3 bytes) 2024-04-24T00:34:27.077 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:27.072+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:27.079 INFO:teuthology.orchestra.run.smithi064.stdout:1160 punch from 0x7f4994a to 0x7f4dff8, (0x46ae bytes) 2024-04-24T00:34:27.079 INFO:teuthology.orchestra.run.smithi064.stdout:1161 writesame 0xb0e5d75 thru 0xb0e7df8 data_size 0x2(0x2084 bytes) 2024-04-24T00:34:27.086 INFO:teuthology.orchestra.run.smithi064.stdout:1162 write 0xce3cb5 thru 0xcf018c (0xc4d8 bytes) 2024-04-24T00:34:27.094 INFO:teuthology.orchestra.run.smithi064.stdout:1163 writesame 0x18d2b81 thru 0x18d7f35 data_size 0x3(0x53b5 bytes) 2024-04-24T00:34:27.128 INFO:teuthology.orchestra.run.smithi064.stdout:1164 write 0xba6f9a7 thru 0xba7218e (0x27e8 bytes) 2024-04-24T00:34:27.134 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:27.134 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:27.134 INFO:teuthology.orchestra.run.smithi064.stdout:1169 write 0x3d4c96 thru 0x3d5a35 (0xda0 bytes) 2024-04-24T00:34:27.137 INFO:teuthology.orchestra.run.smithi064.stdout:1173 read 0x609d8b0 thru 0x60a8af0 (0xb241 bytes) 2024-04-24T00:34:27.137 INFO:teuthology.orchestra.run.smithi064.stdout:1174 write 0x51028c9 thru 0x51097b4 (0x6eec bytes) 2024-04-24T00:34:27.155 INFO:teuthology.orchestra.run.smithi064.stdout:1175 trunc from 0xde063f6 to 0x97fa639 2024-04-24T00:34:27.175 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:27.172+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:27.177 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:27.177 INFO:teuthology.orchestra.run.smithi064.stdout:1179 read 0x7c150ed thru 0x7c16495 (0x13a9 bytes) 2024-04-24T00:34:27.193 INFO:teuthology.orchestra.run.smithi064.stdout:1180 trunc from 0x97fa639 to 0x2446c8e 2024-04-24T00:34:27.205 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:34:27.203+0000 7f841a5f8640 -1 mon.a@0(leader).osd e52 definitely_dead 0 2024-04-24T00:34:27.212 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:27.212 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:27.212 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:27.212 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:27.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:27.239 INFO:teuthology.orchestra.run.smithi064.stdout:1181 trunc from 0x2446c8e to 0xbb92334 2024-04-24T00:34:27.249 INFO:teuthology.orchestra.run.smithi064.stdout:1182 write 0x6c974d thru 0x6cb617 (0x1ecb bytes) 2024-04-24T00:34:27.252 INFO:teuthology.orchestra.run.smithi064.stdout:1183 write 0x419333 thru 0x41c0ba (0x2d88 bytes) 2024-04-24T00:34:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:1184 read 0x4b96aa8 thru 0x4b9c127 (0x5680 bytes) 2024-04-24T00:34:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:1185 write 0xa47d4df thru 0xa48b7a7 (0xe2c9 bytes) 2024-04-24T00:34:27.269 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:27.269 INFO:teuthology.orchestra.run.smithi064.stdout:1187 read 0x3969d69 thru 0x396eeb3 (0x514b bytes) 2024-04-24T00:34:27.269 INFO:teuthology.orchestra.run.smithi064.stdout:1188 write 0x770274c thru 0x770536f (0x2c24 bytes) 2024-04-24T00:34:27.276 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:27.271+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:27.276 INFO:teuthology.orchestra.run.smithi064.stdout:1189 read 0xa291701 thru 0xa29cf4e (0xb84e bytes) 2024-04-24T00:34:27.276 INFO:teuthology.orchestra.run.smithi064.stdout:1190 read 0x9b38ed8 thru 0x9b397b2 (0x8db bytes) 2024-04-24T00:34:27.276 INFO:teuthology.orchestra.run.smithi064.stdout:1191 read 0x40f50cc thru 0x40f89b0 (0x38e5 bytes) 2024-04-24T00:34:27.277 INFO:teuthology.orchestra.run.smithi064.stdout:1192 write 0x4895f58 thru 0x48a56a1 (0xf74a bytes) 2024-04-24T00:34:27.290 INFO:teuthology.orchestra.run.smithi064.stdout:1193 read 0x5fb23f5 thru 0x5fbaf46 (0x8b52 bytes) 2024-04-24T00:34:27.290 INFO:teuthology.orchestra.run.smithi064.stdout:1194 read 0xaacb5c thru 0xaba849 (0xdcee bytes) 2024-04-24T00:34:27.291 INFO:teuthology.orchestra.run.smithi064.stdout:1195 writesame 0x7176fcc thru 0x717c74e data_size 0x1(0x5783 bytes) 2024-04-24T00:34:27.359 INFO:teuthology.orchestra.run.smithi064.stdout:1197 trunc from 0xbb92334 to 0x740b512 2024-04-24T00:34:27.376 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:27.371+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:27.381 INFO:teuthology.orchestra.run.smithi064.stdout:1201 write 0x546480d thru 0x546b5e6 (0x6dda bytes) 2024-04-24T00:34:27.389 INFO:teuthology.orchestra.run.smithi064.stdout:1202 punch from 0x2c34307 to 0x2c3eab0, (0xa7a9 bytes) 2024-04-24T00:34:27.400 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:27.400 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:27.400 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:27.400 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:27.400 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:27.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:27.420 INFO:teuthology.orchestra.run.smithi064.stdout:1203 write 0xeddf016 thru 0xedec2f9 (0xd2e4 bytes) 2024-04-24T00:34:27.425 INFO:teuthology.orchestra.run.smithi064.stdout:1207 write 0x5b806b5 thru 0x5b81ba4 (0x14f0 bytes) 2024-04-24T00:34:27.432 INFO:teuthology.orchestra.run.smithi064.stdout:1208 writesame 0xd2a045c thru 0xd2b031a data_size 0x1(0xfebf bytes) 2024-04-24T00:34:27.477 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:27.472+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:27.538 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3338 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13434880~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.706639006, 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021864103999999999, 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.539 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3338 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828603+0000", 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3699999999999999e-07 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828608+0000", 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.216e-06 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828619+0000", 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.172e-05 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828627+0000", 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.79e-06 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850192+0000", 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021564967000000001 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850314+0000", 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121789 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850467+0000", 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015268500000000001 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3339 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13565952~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.706603343000001, 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022319400999999999, 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3339 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828639+0000", 2024-04-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0100000000000001e-06 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828643+0000", 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2300000000000002e-06 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828652+0000", 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000001e-06 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828659+0000", 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850539+0000", 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021879770999999999 2024-04-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850647+0000", 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010767 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850958+0000", 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031074100000000001 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3400 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13828096~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.699806923000001, 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022009407000000002, 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.549 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3400 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835435+0000", 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835436+0000", 2024-04-24T00:34:27.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8139999999999999e-06 2024-04-24T00:34:27.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835462+0000", 2024-04-24T00:34:27.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5273e-05 2024-04-24T00:34:27.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835489+0000", 2024-04-24T00:34:27.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6859000000000002e-05 2024-04-24T00:34:27.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835507+0000", 2024-04-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8144000000000001e-05 2024-04-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.856821+0000", 2024-04-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021313959 2024-04-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857070+0000", 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024914099999999998 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857444+0000", 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037421699999999998 2024-04-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3401 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13959168~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.699710447000001, 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022623199, 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.556 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3401 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835532+0000", 2024-04-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3769999999999999e-06 2024-04-24T00:34:27.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835540+0000", 2024-04-24T00:34:27.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2969999999999998e-06 2024-04-24T00:34:27.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835559+0000", 2024-04-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9199999999999999e-05 2024-04-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835574+0000", 2024-04-24T00:34:27.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4983e-05 2024-04-24T00:34:27.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857648+0000", 2024-04-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022074522999999999 2024-04-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857816+0000", 2024-04-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016745200000000001 2024-04-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858154+0000", 2024-04-24T00:34:27.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033836700000000001 2024-04-24T00:34:27.565 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3402 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14090240~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.699647587000001, 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023133308000000002, 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3402 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835595+0000", 2024-04-24T00:34:27.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.043e-06 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835602+0000", 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7329999999999997e-06 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835624+0000", 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2079999999999999e-05 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835638+0000", 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4270000000000001e-05 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858219+0000", 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022580586 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858442+0000", 2024-04-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000223191 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858727+0000", 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028540500000000002 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3403 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14221312~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.699584171000001, 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023734209999999999, 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3403 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835658+0000", 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.083e-06 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835664+0000", 2024-04-24T00:34:27.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5999999999999997e-06 2024-04-24T00:34:27.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835683+0000", 2024-04-24T00:34:27.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8604e-05 2024-04-24T00:34:27.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835697+0000", 2024-04-24T00:34:27.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4576000000000001e-05 2024-04-24T00:34:27.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858838+0000", 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023140524999999999 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859063+0000", 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000225598 2024-04-24T00:34:27.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859392+0000", 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032822400000000001 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3404 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14352384~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.699525621999999, 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024301263, 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.577 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3404 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835717+0000", 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835760+0000", 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2993e-05 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835781+0000", 2024-04-24T00:34:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0743000000000001e-05 2024-04-24T00:34:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835796+0000", 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5233e-05 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859493+0000", 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023697296999999999 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859722+0000", 2024-04-24T00:34:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000228619 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860017+0000", 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029529399999999998 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3405 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14483456~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.699422272, 2024-04-24T00:34:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024591302999999998, 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3405 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835821+0000", 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835827+0000", 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5300000000000002e-06 2024-04-24T00:34:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835845+0000", 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8096000000000001e-05 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835859+0000", 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3916e-05 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860117+0000", 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024257616999999999 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860241+0000", 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012400200000000001 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860411+0000", 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169562 2024-04-24T00:34:27.586 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3406 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14614528~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.699361963000001, 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024976411, 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3406 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835881+0000", 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835887+0000", 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3369999999999996e-06 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835905+0000", 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7592999999999999e-05 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835919+0000", 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4277000000000001e-05 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860500+0000", 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024580722999999999 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860651+0000", 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015075900000000001 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860856+0000", 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020540200000000001 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3455 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.696542440999998, 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022775208000000002, 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3455 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838702+0000", 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838715+0000", 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2959000000000001e-05 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838740+0000", 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4893e-05 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838753+0000", 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3176999999999999e-05 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860986+0000", 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022233541999999999 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.861203+0000", 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000216472 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.861474+0000", 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027145799999999997 2024-04-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.894503485000001, 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.894427285999999, 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.893969112000001, 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:27.609 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.893911804999998, 2024-04-24T00:34:27.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:34:27.610 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.610 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.610 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.610 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.610 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.610 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.611 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:34:27.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.611 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:27.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:27.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:34:27.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:34:27.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:34:27.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:34:27.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:34:27.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:34:27.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:34:27.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:34:27.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:34:27.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:34:27.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:34:27.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:34:27.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:34:27.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:34:27.618 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.618 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.618 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.893652839000001, 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.620 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.620 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.620 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:34:27.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.620 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:27.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:27.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:34:27.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:34:27.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:34:27.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:34:27.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:34:27.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:34:27.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:34:27.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:34:27.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:34:27.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:34:27.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:34:27.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:34:27.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:34:27.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:34:27.627 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.627 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.627 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.628 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:27.628 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:27.628 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.892731367, 2024-04-24T00:34:27.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:34:27.628 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.629 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.629 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.629 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:34:27.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:27.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:27.633 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.320916637, 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:34:27.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:34:27.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:34:27.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.305387388, 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.639 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.676664068000001, 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.643 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:34:27.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:34:27.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:34:27.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:34:27.648 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.661628941, 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.652 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:27.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:34:27.654 INFO:teuthology.orchestra.run.smithi064.stdout:1210 read 0x85c7d9e thru 0x85d2737 (0xa99a bytes) 2024-04-24T00:34:27.654 INFO:teuthology.orchestra.run.smithi064.stdout:1212 writesame 0x8900b21 thru 0x8910028 data_size 0x4(0xf508 bytes) 2024-04-24T00:34:27.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:27.656 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:27.651+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:27.716 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.5. 2024-04-24T00:34:27.740 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-24T00:34:27.755 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:27.751+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:27.827 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3.b516dbdd (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:34:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.11699757, 2024-04-24T00:34:27.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11702028, 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:34:27.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3.2aad49d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.116899645, 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116968586, 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:27.831 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:27.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:34:27.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:34:27.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:34:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:34:27.835 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 2 2024-04-24T00:34:27.835 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:27.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:27.856 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:27.851+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:27.957 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:27.952+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:27.966 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:27.966 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:27.966 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:27.966 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:27.966 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:27.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:28.067 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:28.064+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:28.144 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:28.144 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:28.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:28.144 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [] 2024-04-24T00:34:28.144 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:28.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:28.199 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:28.196+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:28.259 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:28.256+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:28.315 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:28.315 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:28.315 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:28.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:28.360 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:28.355+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:28.461 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:28.459+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:28.495 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:28.496 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:28.496 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:28.496 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:28.496 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:28.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:28.562 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:28.556+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:5336 3.1d e36/21 3:b83bad9e:::rbd_data.115ed6c33904.0000000000000086:head v 36'3, mlcod=34'1)", 2024-04-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.936062024000002, 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010615080000000001, 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694760+0000", 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8080000000000001e-06 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694891+0000", 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013035600000000001 2024-04-24T00:34:28.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694916+0000", 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5392e-05 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694930+0000", 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4059999999999999e-05 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695055+0000", 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124499 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695112+0000", 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7853000000000003e-05 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705273+0000", 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010160626000000001 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705373+0000", 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100486 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10257 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.428418065000001, 2024-04-24T00:34:28.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011690176, 2024-04-24T00:34:28.643 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.643 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.643 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.643 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10257 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202404+0000", 2024-04-24T00:34:28.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.897e-06 2024-04-24T00:34:28.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202412+0000", 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202430+0000", 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8043e-05 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202442+0000", 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439999999999999e-05 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207283+0000", 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048410600000000003 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207552+0000", 2024-04-24T00:34:28.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026864100000000002 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.208755+0000", 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012028189999999999 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.211687+0000", 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029325240000000002 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.213944+0000", 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022567300000000002 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214023+0000", 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8932999999999998e-05 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214093+0000", 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9538999999999999e-05 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10262 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.338074993999999, 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089239360000000004, 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.650 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10262 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292747+0000", 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.083e-06 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292753+0000", 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2700000000000001e-06 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292767+0000", 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4175999999999999e-05 2024-04-24T00:34:28.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292776+0000", 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0939999999999997e-06 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.295605+0000", 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028292909999999998 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.295896+0000", 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000290809 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296379+0000", 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048252899999999999 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.299342+0000", 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002962957 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301530+0000", 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021886259999999999 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301605+0000", 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4893000000000003e-05 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301669+0000", 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4208000000000006e-05 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10263 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.337982778000001, 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091981449999999996, 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10263 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:28.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292839+0000", 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-06 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292846+0000", 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.81e-06 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292862+0000", 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6456e-05 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292870+0000", 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3699999999999995e-06 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296812+0000", 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039413900000000003 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296881+0000", 2024-04-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9426999999999998e-05 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.297166+0000", 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028513300000000002 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301682+0000", 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045159029999999999 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301927+0000", 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024475499999999998 2024-04-24T00:34:28.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301980+0000", 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2796000000000001e-05 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.302036+0000", 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5992000000000002e-05 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.661 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10288 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.298074651, 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.009564744, 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10288 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332747+0000", 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332769+0000", 2024-04-24T00:34:28.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1903e-05 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332780+0000", 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1243000000000001e-05 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332789+0000", 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0699999999999996e-06 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336540+0000", 2024-04-24T00:34:28.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037510339999999999 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336760+0000", 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021991099999999999 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.337418+0000", 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000658127 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.339101+0000", 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016826390000000001 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342150+0000", 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030494850000000002 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342236+0000", 2024-04-24T00:34:28.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6212999999999996e-05 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342311+0000", 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4228999999999997e-05 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10359 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.187640246999999, 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091612789999999996, 2024-04-24T00:34:28.667 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10359 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443182+0000", 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443189+0000", 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1470000000000004e-06 2024-04-24T00:34:28.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443206+0000", 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7603000000000001e-05 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443221+0000", 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4747e-05 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.448444+0000", 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052232700000000003 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.448605+0000", 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016098499999999999 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.449528+0000", 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092301199999999996 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451989+0000", 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024603250000000002 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452174+0000", 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018491100000000001 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452265+0000", 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1283e-05 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452341+0000", 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6656000000000006e-05 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10365 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.136048432000001, 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098439340000000004, 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.673 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10365 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494773+0000", 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1399999999999995e-07 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494782+0000", 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3649999999999993e-06 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494794+0000", 2024-04-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1606e-05 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494804+0000", 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0013999999999999e-05 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498700+0000", 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038961310000000002 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498969+0000", 2024-04-24T00:34:28.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000269432 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.499771+0000", 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080119200000000005 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.503502+0000", 2024-04-24T00:34:28.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037318300000000002 2024-04-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504304+0000", 2024-04-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080115000000000004 2024-04-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504475+0000", 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017146199999999999 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504616+0000", 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014083800000000001 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.679 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10409 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.054466230999999, 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010506112, 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10409 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576355+0000", 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576359+0000", 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3229999999999999e-06 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576372+0000", 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576383+0000", 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1192999999999999e-05 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580539+0000", 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041557230000000001 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580813+0000", 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027430400000000001 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.582037+0000", 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001223879 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.584201+0000", 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002164408 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586567+0000", 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023657090000000001 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586731+0000", 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016439800000000001 2024-04-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586860+0000", 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128835 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10518 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.8813938, 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091784659999999997, 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.685 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10518 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2699999999999998e-07 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749432+0000", 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2780000000000004e-06 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749444+0000", 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1882000000000001e-05 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749451+0000", 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1890000000000001e-06 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.753811+0000", 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043599609999999999 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.753978+0000", 2024-04-24T00:34:28.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016679100000000001 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.754514+0000", 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00053602499999999998 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758097+0000", 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035836169999999999 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758429+0000", 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033133099999999998 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758527+0000", 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8919000000000006e-05 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758605+0000", 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7645999999999995e-05 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.690 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10645 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.603462149, 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010437706999999999, 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10645 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.691 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027359+0000", 2024-04-24T00:34:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.06e-06 2024-04-24T00:34:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027365+0000", 2024-04-24T00:34:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2769999999999996e-06 2024-04-24T00:34:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027380+0000", 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5537e-05 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027389+0000", 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5299999999999996e-06 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.030838+0000", 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003449232 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031115+0000", 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000277254 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031925+0000", 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080932599999999997 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.035618+0000", 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003693647 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037627+0000", 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020091390000000001 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037720+0000", 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2775000000000005e-05 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037796+0000", 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.593e-05 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10664 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.567193898999999, 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088645240000000004, 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.697 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10664 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063633+0000", 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5230000000000003e-06 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063645+0000", 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1725999999999999e-05 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063653+0000", 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6500000000000002e-06 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.068370+0000", 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047163049999999996 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.068639+0000", 2024-04-24T00:34:28.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026901399999999998 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.069434+0000", 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00079544299999999995 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.071514+0000", 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020798180000000002 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072319+0000", 2024-04-24T00:34:28.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080491299999999998 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072413+0000", 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4668999999999998e-05 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072491+0000", 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7589000000000005e-05 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10719 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.452290981999999, 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010945605000000001, 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10719 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178530+0000", 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1800000000000005e-07 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178535+0000", 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5129999999999998e-06 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178544+0000", 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2359999999999993e-06 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178550+0000", 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.421e-06 2024-04-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181685+0000", 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031346989999999999 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181965+0000", 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027960399999999997 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.182619+0000", 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065424799999999996 2024-04-24T00:34:28.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.186425+0000", 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038062690000000001 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189312+0000", 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028871809999999999 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189407+0000", 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4345000000000006e-05 2024-04-24T00:34:28.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189475+0000", 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8270999999999997e-05 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10739 3.10 3:0a996d09:::rbd_data.115e380d621c.0000000000000199:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.40939448, 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092681780000000002, 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.708 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10739 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:28.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221428+0000", 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0949999999999998e-06 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221438+0000", 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9310000000000003e-06 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221463+0000", 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5023000000000001e-05 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221477+0000", 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3591000000000001e-05 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221633+0000", 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156565 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.222026+0000", 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039234399999999998 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.223749+0000", 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017237109999999999 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230482+0000", 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067323670000000004 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230517+0000", 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981999999999998e-05 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230610+0000", 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230694+0000", 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4393000000000003e-05 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.713 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10768 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.348163575999999, 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092323920000000007, 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10768 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282658+0000", 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1799999999999999e-06 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282664+0000", 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6339999999999998e-06 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282676+0000", 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2320000000000001e-05 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282687+0000", 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1103e-05 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284651+0000", 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00196408 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284827+0000", 2024-04-24T00:34:28.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175768 2024-04-24T00:34:28.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.285720+0000", 2024-04-24T00:34:28.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089352799999999999 2024-04-24T00:34:28.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289504+0000", 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003783813 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291726+0000", 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022220339999999999 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291814+0000", 2024-04-24T00:34:28.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7425999999999999e-05 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291889+0000", 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5506000000000005e-05 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10769 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:28.720 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.348078999, 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012818598, 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10769 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282742+0000", 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282747+0000", 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0329999999999999e-06 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282759+0000", 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1687e-05 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282768+0000", 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.286885+0000", 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041171139999999998 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.287035+0000", 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149968 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.288104+0000", 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010690540000000001 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295331+0000", 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072273800000000003 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295418+0000", 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295493+0000", 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4825000000000003e-05 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295560+0000", 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7191000000000001e-05 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10770 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.34803975, 2024-04-24T00:34:28.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013929258999999999, 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10770 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282782+0000", 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9000000000000005e-07 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282786+0000", 2024-04-24T00:34:28.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1330000000000001e-06 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282795+0000", 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-06 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282803+0000", 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8499999999999994e-06 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289625+0000", 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068213689999999999 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289770+0000", 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014539899999999999 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.290476+0000", 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070591399999999998 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295352+0000", 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048766089999999996 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296546+0000", 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011937580000000001 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296638+0000", 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2179e-05 2024-04-24T00:34:28.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296710+0000", 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1527999999999997e-05 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.34800665, 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:34:28.732 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:34:28.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:34:28.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:34:28.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:28.738 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.347970637, 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:34:28.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:34:28.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:34:28.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.098566243, 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:28.744 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:28.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:34:28.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:34:28.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:34:28.750 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.750 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.750 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.750 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:34:28.750 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.06452919, 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:28.754 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:28.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:28.757 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:34:28.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:28.758 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:28.751+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:28.760 INFO:teuthology.orchestra.run.smithi064.stdout:1213 trunc from 0xedec2fa to 0x5e5dfb8 2024-04-24T00:34:28.818 INFO:teuthology.orchestra.run.smithi064.stdout:1215 write 0x7367208 thru 0x73691bd (0x1fb6 bytes) 2024-04-24T00:34:28.825 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:28.825 INFO:teuthology.orchestra.run.smithi064.stdout:1218 write 0x80d9cd thru 0x8191fe (0xb832 bytes) 2024-04-24T00:34:28.830 INFO:teuthology.orchestra.run.smithi064.stdout:1221 punch from 0x1c11231 to 0x1c1d8a0, (0xc66f bytes) 2024-04-24T00:34:28.830 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:28.830 INFO:teuthology.orchestra.run.smithi064.stdout:1223 read 0x616555 thru 0x623861 (0xd30d bytes) 2024-04-24T00:34:28.856 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:28.852+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:28.913 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:28.914 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:28.930 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:28.930 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:28.937 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:28.937 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:28.937 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:28.937 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:28.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:28.947 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:28.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:28.957 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:28.955+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:28.964 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:28.965 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:28.983 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:28.983 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:29.001 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:29.001 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:29.019 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:29.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:29.036 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:29.036 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:29.058 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:29.052+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:29.116 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:29.117 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:29.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:29.175 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:29.172+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:29.260 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:29.255+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:29.296 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:29.297 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:29.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:29.360 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:29.355+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:29.446 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:29.446 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:29.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:29.461 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:29.456+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:29.563 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:29.560+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:29.625 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:29.626 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:29.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:29.663 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:29.660+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:29.765 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:29.760+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:29.770 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:29.770 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:29.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:29.866 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:29.863+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:29.898 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:29.898 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:29.898 INFO:teuthology.orchestra.run.smithi064.stdout:1226 clone 8 order 24 su 524288 sc 14 2024-04-24T00:34:29.915 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:29.915 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:29.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:29.973 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:29.971+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:30.067 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:30.063+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:30.104 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:30.104 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:30.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:30.209 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:30.207+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:30.269 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:30.264+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:30.290 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:30.290 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:30.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:30.389 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:30.387+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:30.472 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:30.467+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:30.472 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:30.472 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:30.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:30.572 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:30.568+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:30.647 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:30.648 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:30.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:30.705 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:30.703+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:30.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:34:30.787 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:30.787 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:30.788 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:30.783+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:30.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:30.875 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:30.872+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:30.967 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:30.967 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:30.976 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:30.972+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:30.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:31.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:31.099 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:31.099 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:31.099 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:31.095+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:31.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:31.193 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:31.193 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:31.193 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:31.191+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:31.210 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:31.210 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:31.227 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:31.227 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:31.243 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:31.243 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:31.263 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:31.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:31.278 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:31.275+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:31.280 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2036 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.747534476999999, 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020639231000000001, 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2036 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528464+0000", 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600000000000004e-07 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528467+0000", 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8040000000000002e-06 2024-04-24T00:34:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528474+0000", 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7270000000000007e-06 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528480+0000", 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3340000000000001e-06 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548902+0000", 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020421963000000001 2024-04-24T00:34:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548991+0000", 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9332000000000005e-05 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549102+0000", 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111535 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2053 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.747274306000001, 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616257999999998, 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2053 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528724+0000", 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6400000000000002e-07 2024-04-24T00:34:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528727+0000", 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6749999999999998e-06 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528734+0000", 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4739999999999999e-06 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528740+0000", 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4759999999999997e-06 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549138+0000", 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020398315 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549225+0000", 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6802e-05 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549340+0000", 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011495200000000001 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2054 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.747250439999998, 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020836107, 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2054 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528748+0000", 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999999e-07 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528750+0000", 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528756+0000", 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1249999999999998e-06 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528761+0000", 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549377+0000", 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616254000000001 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549468+0000", 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1199000000000006e-05 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549583+0000", 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011484099999999999 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2055 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.74722946, 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022112625, 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2055 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528769+0000", 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9299999999999999e-07 2024-04-24T00:34:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528771+0000", 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2809999999999998e-06 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528777+0000", 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.127e-06 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528782+0000", 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8330000000000001e-06 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550648+0000", 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021866450999999999 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550755+0000", 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010676800000000001 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550881+0000", 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012577199999999999 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2056 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.747207742000001, 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022390911999999999, 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2056 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528792+0000", 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1390000000000002e-06 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528798+0000", 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9869999999999996e-06 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528804+0000", 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4949999999999999e-06 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550928+0000", 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022124061 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551050+0000", 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121981 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551181+0000", 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130858 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.747186797000001, 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:34:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:34:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.747162883000001, 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:34:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:34:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.747135708999998, 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:34:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:34:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.747109603999998, 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:34:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.746698623, 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:34:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.746678813999999, 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:34:31.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.746625755, 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.326 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:34:31.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:34:31.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:34:31.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.746590899000001, 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:34:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:34:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.746569335, 2024-04-24T00:34:31.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:34:31.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:34:31.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.746539843000001, 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.339 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.746518981000001, 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.343 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:31.347 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.746496406999999, 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:34:31.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.351 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.064236909000002, 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.352 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:34:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:34:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:34:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.063885994, 2024-04-24T00:34:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:34:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:34:31.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.362 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.056997674000002, 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:31.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:34:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:31.367 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:31.367 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:31.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:31.369 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:31.409 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:31.407+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:31.480 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:31.476+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:31.499 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:31.499 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:31.499 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:31.499 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:31.513 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:31.531 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:31.599 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:31.596+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:31.677 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:31.677 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:31.677 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:31.678 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:31.678 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:31.681 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:31.679+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:31.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:31.740 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:34:31.782 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:31.776+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:31.858 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:31.858 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:31.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:31.858 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3339 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13565952~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.024392088999999, 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022319400999999999, 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3339 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828638+0000", 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828639+0000", 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0100000000000001e-06 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828643+0000", 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2300000000000002e-06 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828652+0000", 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000001e-06 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.828659+0000", 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1389999999999998e-06 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850539+0000", 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021879770999999999 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850647+0000", 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010767 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.850958+0000", 2024-04-24T00:34:31.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031074100000000001 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3401 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 13959168~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.017499192999999, 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022623199, 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3401 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835531+0000", 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835532+0000", 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3769999999999999e-06 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835540+0000", 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2969999999999998e-06 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835559+0000", 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9199999999999999e-05 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835574+0000", 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4983e-05 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857648+0000", 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022074522999999999 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.857816+0000", 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016745200000000001 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858154+0000", 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033836700000000001 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3402 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14090240~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.017436332999999, 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023133308000000002, 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3402 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835594+0000", 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835595+0000", 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.043e-06 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835602+0000", 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7329999999999997e-06 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835624+0000", 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2079999999999999e-05 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835638+0000", 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4270000000000001e-05 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858219+0000", 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022580586 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858442+0000", 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000223191 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858727+0000", 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028540500000000002 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3403 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14221312~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.017372916999999, 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023734209999999999, 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3403 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835657+0000", 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835658+0000", 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.083e-06 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835664+0000", 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5999999999999997e-06 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835683+0000", 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8604e-05 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835697+0000", 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4576000000000001e-05 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.858838+0000", 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023140524999999999 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859063+0000", 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000225598 2024-04-24T00:34:31.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859392+0000", 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032822400000000001 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3404 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14352384~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.017314368000001, 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024301263, 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.875 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3404 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835716+0000", 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835717+0000", 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835760+0000", 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2993e-05 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835781+0000", 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0743000000000001e-05 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835796+0000", 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5233e-05 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859493+0000", 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023697296999999999 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.859722+0000", 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000228619 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860017+0000", 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029529399999999998 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.879 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3405 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14483456~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.017211018000001, 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024591302999999998, 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3405 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835819+0000", 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835821+0000", 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835827+0000", 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5300000000000002e-06 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835845+0000", 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8096000000000001e-05 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835859+0000", 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3916e-05 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860117+0000", 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024257616999999999 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860241+0000", 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012400200000000001 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860411+0000", 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169562 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.883 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3406 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:5 [sparse-read 14614528~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.017150708999999, 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024976411, 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.884 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3406 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835880+0000", 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835881+0000", 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835887+0000", 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3369999999999996e-06 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835905+0000", 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7592999999999999e-05 2024-04-24T00:34:31.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.835919+0000", 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4277000000000001e-05 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860500+0000", 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024580722999999999 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860651+0000", 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015075900000000001 2024-04-24T00:34:31.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860856+0000", 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020540200000000001 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:3455 3.9 3:9063950d:::rbd_data.115eb9c76af6.0000000000000000:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.014331187, 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022775208000000002, 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.888 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 3455 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838699+0000", 2024-04-24T00:34:31.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838702+0000", 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838715+0000", 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2959000000000001e-05 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838740+0000", 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4893e-05 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.838753+0000", 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3176999999999999e-05 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.860986+0000", 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022233541999999999 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.861203+0000", 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000216472 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:04.861474+0000", 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027145799999999997 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.892 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.212292230999999, 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.212216032000001, 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:34:31.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.211757857999999, 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:34:31.901 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.211700551, 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:34:31.905 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:31.909 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.211441584999999, 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:34:31.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.913 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.210520113000001, 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.914 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:34:31.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.638705383000001, 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:34:31.918 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.923 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.623176134000001, 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:34:31.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:34:31.927 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.994452813999999, 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:34:31.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:34:31.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:34:31.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:34:31.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.979417687000002, 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.933 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.216919689, 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:34:31.937 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:34:31.942 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.2030967309999996, 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:34:31.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:31.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:31.946 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:31.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:31.948 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:31.944+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:32.048 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:32.047+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:32.118 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:32.118 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:32.119 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:32.119 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:32.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:32.149 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:32.147+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:32.250 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:32.247+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:32.251 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:32.251 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:32.251 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:32.252 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:32.252 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:32.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:32.351 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:32.348+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:32.388 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:32.388 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:32.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:32.388 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:32.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.388 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.6763819990000002, 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:34:32.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:34:32.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.394 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.6762840739999998, 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:34:32.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:34:32.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.1d 54/53 [PushOp(3:b92b5540:::rbd_header.115eeb9272a5:head, version: 52'55, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 10, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:b92b5540:::rbd_header.115eeb9272a5:head@52'55, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.733051+0000", 2024-04-24T00:34:32.401 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6516993530000001, 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033708420000000002, 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.733051+0000", 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.733051+0000", 2024-04-24T00:34:32.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.733051+0000", 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5700000000000002e-07 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.733075+0000", 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3309999999999999e-05 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.733099+0000", 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4652e-05 2024-04-24T00:34:32.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.733136+0000", 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6259e-05 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.735181+0000", 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020458970000000001 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.735193+0000", 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1165000000000001e-05 2024-04-24T00:34:32.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736421+0000", 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001228802 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.1f 54/53 [PushOp(3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:head, version: 30'19, data_included: [4221637~41759,15086854~35619,16503620~3641], data_size: 81019, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:head@30'19, size: 16507261, copy_subset: [(0, 16507261)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 16507261, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.737253+0000", 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6474964750000001, 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059692310000000002, 2024-04-24T00:34:32.405 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737253+0000", 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737253+0000", 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737254+0000", 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.004e-06 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737384+0000", 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012957800000000001 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737405+0000", 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1010999999999999e-05 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737420+0000", 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5201000000000001e-05 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737626+0000", 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020537299999999999 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737662+0000", 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5966000000000001e-05 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.743223+0000", 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055610980000000004 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.2 54/53 [PushOp(3:46b217ad:::rbd_data.115eb9c76af6.0000000000000005:head, version: 30'22, data_included: [5974226~52932,7593401~3579], data_size: 56511, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:46b217ad:::rbd_data.115eb9c76af6.0000000000000005:head@30'22, size: 15728640, copy_subset: [(0, 15728640)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 15728640, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.740070+0000", 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6446796150000003, 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039967989999999997, 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.740070+0000", 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.740070+0000", 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.740071+0000", 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5499999999999996e-07 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.740153+0000", 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2106000000000003e-05 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.740184+0000", 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1143000000000003e-05 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.740224+0000", 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9529e-05 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.740362+0000", 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013844500000000001 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.740376+0000", 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3492999999999999e-05 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.744067+0000", 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036911280000000001 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.2 54/53 [PushOp(3:40b32418:::rbd_data.115e380d621c.000000000000000b:head, version: 40'31, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:40b32418:::rbd_data.115e380d621c.000000000000000b:head@40'31, size: 262144, copy_subset: [(0, 262144)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262144, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.749934+0000", 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6348161939999999, 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029155280000000001, 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.749934+0000", 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.749934+0000", 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.749935+0000", 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.77e-06 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.750576+0000", 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064040699999999996 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.750615+0000", 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9329999999999998e-05 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.750653+0000", 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7494999999999999e-05 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.750827+0000", 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017442299999999999 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.750839+0000", 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2296000000000001e-05 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.752849+0000", 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020098070000000002 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.2 54/53 [PushOp(3:4506e383:::rbd_data.115e380d621c.0000000000000130:head, version: 41'37, data_included: [0~16384], data_size: 16384, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4506e383:::rbd_data.115e380d621c.0000000000000130:head@41'37, size: 16384, copy_subset: [(0, 16384)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 16384, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.760970+0000", 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6237803409999998, 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027176980000000002, 2024-04-24T00:34:32.417 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.760970+0000", 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.760970+0000", 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.760971+0000", 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0020000000000001e-06 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.761008+0000", 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7132000000000003e-05 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.761036+0000", 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8564e-05 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.761063+0000", 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7163999999999999e-05 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.761275+0000", 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021133799999999999 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.761295+0000", 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0049999999999999e-05 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.763687+0000", 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023924480000000001 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21651 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.762611+0000", 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6221393759999998, 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027645320000000001, 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21651 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.762611+0000", 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.762611+0000", 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.762613+0000", 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4959999999999999e-06 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.762636+0000", 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3300000000000001e-05 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.762670+0000", 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3877000000000001e-05 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.762694+0000", 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3793000000000001e-05 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.762816+0000", 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012232700000000001 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.762969+0000", 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152576 2024-04-24T00:34:32.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.763222+0000", 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000252988 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.764869+0000", 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001647445 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.765158+0000", 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028896800000000002 2024-04-24T00:34:32.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.765232+0000", 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3918999999999995e-05 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.765375+0000", 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000142843 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:21650 3.d e54/53 3:b4db5745:::rbd_data.115eeb9272a5.000000000000001f:head v 54'18, mlcod=0'0)", 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.766463+0000", 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6182865949999998, 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058822060000000001, 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.766463+0000", 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.766463+0000", 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.766464+0000", 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5799999999999998e-07 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.766471+0000", 2024-04-24T00:34:32.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8889999999999996e-06 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.766483+0000", 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1467e-05 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.766490+0000", 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0609999999999998e-06 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.766532+0000", 2024-04-24T00:34:32.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2660999999999997e-05 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.766548+0000", 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5551999999999999e-05 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.772252+0000", 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057040490000000001 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.772345+0000", 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3568999999999998e-05 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21656 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.777387+0000", 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.6073633510000001, 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036081989999999999, 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.431 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21656 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.777387+0000", 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.777387+0000", 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.777388+0000", 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3960000000000001e-06 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.777394+0000", 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5849999999999999e-06 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.777407+0000", 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3081000000000001e-05 2024-04-24T00:34:32.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.777414+0000", 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8790000000000003e-06 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.777527+0000", 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011227599999999999 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.777747+0000", 2024-04-24T00:34:32.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000220564 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.778197+0000", 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044968800000000001 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.780854+0000", 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026569010000000001 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:32.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.780873+0000", 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9001e-05 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.780921+0000", 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8038000000000003e-05 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.780995+0000", 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3789999999999997e-05 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.1d 54/53 [PushOp(3:b83bad9e:::rbd_data.115ed6c33904.0000000000000086:head, version: 36'3, data_included: [579975~54593], data_size: 54593, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:b83bad9e:::rbd_data.115ed6c33904.0000000000000086:head@36'3, size: 634568, copy_subset: [(0, 634568)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 634568, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.786112+0000", 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.59863752, 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049393709999999997, 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.786112+0000", 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.786112+0000", 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.786114+0000", 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3319999999999999e-06 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.786227+0000", 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113337 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.786272+0000", 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5167000000000003e-05 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.786310+0000", 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7540000000000003e-05 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.787000+0000", 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00069001100000000003 2024-04-24T00:34:32.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.787011+0000", 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1511999999999999e-05 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791052+0000", 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040404719999999998 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.440 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21661 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [call,call rbd.parent_overlap_get in=21b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.788152+0000", 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.596597552, 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035744430000000001, 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21661 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.441 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788152+0000", 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788152+0000", 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788154+0000", 2024-04-24T00:34:32.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4619999999999999e-06 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788161+0000", 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7580000000000003e-06 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788179+0000", 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7992e-05 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788190+0000", 2024-04-24T00:34:32.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1454000000000001e-05 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791098+0000", 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029082719999999999 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791215+0000", 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116501 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791727+0000", 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051200400000000004 2024-04-24T00:34:32.444 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21662 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5965336720000001, 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082640219999999993, 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.445 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21662 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788217+0000", 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788221+0000", 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0470000000000004e-06 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788234+0000", 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3138e-05 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788243+0000", 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1759999999999993e-06 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791802+0000", 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003559347 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791936+0000", 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134349 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.796480+0000", 2024-04-24T00:34:32.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004544022 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21667 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.801921+0000", 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5828292199999998, 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062889759999999999, 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.449 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21667 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801921+0000", 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801921+0000", 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801923+0000", 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3e-06 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801935+0000", 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2068e-05 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801958+0000", 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3124000000000001e-05 2024-04-24T00:34:32.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801972+0000", 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3485e-05 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.802045+0000", 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2924999999999997e-05 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.802151+0000", 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106289 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.802330+0000", 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017869699999999999 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.804359+0000", 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020295410000000002 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.807858+0000", 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034990049999999999 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.807958+0000", 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010035 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.808210+0000", 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025119200000000002 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.1d 54/53 [PushOp(3:bd25af37:::rbd_data.115e380d621c.0000000000000003:head, version: 39'5, data_included: [88064~161386], data_size: 161386, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bd25af37:::rbd_data.115e380d621c.0000000000000003:head@39'5, size: 249450, copy_subset: [(0, 249450)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 249450, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.804752+0000", 2024-04-24T00:34:32.454 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5799982259999998, 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030858600000000002, 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.804752+0000", 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.804752+0000", 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.804753+0000", 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4339999999999999e-06 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.804977+0000", 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022431299999999999 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.805015+0000", 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7327999999999999e-05 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.805065+0000", 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0581999999999998e-05 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.805176+0000", 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110479 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.805189+0000", 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2982999999999999e-05 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.807838+0000", 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002648741 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21671 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [call,call rbd.parent_overlap_get in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.811899+0000", 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5728507280000001, 2024-04-24T00:34:32.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024727120000000002, 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21671 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.811899+0000", 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.811899+0000", 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.811909+0000", 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6889999999999999e-06 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.811915+0000", 2024-04-24T00:34:32.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6779999999999997e-06 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.811928+0000", 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3852000000000001e-05 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.811936+0000", 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5530000000000004e-06 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.813343+0000", 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001407487 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.813540+0000", 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019637800000000001 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.814372+0000", 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00083207500000000005 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.d 54/53 [PushOp(3:b6c53cc7:::rbd_data.115e380d621c.0000000000000009:head, version: 40'4, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:b6c53cc7:::rbd_data.115e380d621c.0000000000000009:head@40'4, size: 262144, copy_subset: [(0, 262144)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262144, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.815431+0000", 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5693191080000002, 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003310959, 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.815431+0000", 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.815431+0000", 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.815432+0000", 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.468e-06 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.815861+0000", 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00042897500000000001 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.815896+0000", 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5002000000000001e-05 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.815932+0000", 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6050999999999998e-05 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.816077+0000", 2024-04-24T00:34:32.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000145212 2024-04-24T00:34:32.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.816104+0000", 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6021999999999999e-05 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.818742+0000", 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026382290000000002 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.470 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.1b 54/53 [PushOp(3:de44b52c:::rbd_data.115e5ff16b0c.0000000000000002:head, version: 34'14, data_included: [786432~5058,1579855~47210], data_size: 52268, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:de44b52c:::rbd_data.115e5ff16b0c.0000000000000002:head@34'14, size: 1627065, copy_subset: [(0, 1627065)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1627065, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.848983+0000", 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.535766991, 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037815639999999998, 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.848983+0000", 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.848983+0000", 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.848984+0000", 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3799999999999996e-07 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.849011+0000", 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7083000000000001e-05 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.849029+0000", 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8275999999999999e-05 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.849041+0000", 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2318e-05 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.849145+0000", 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103426 2024-04-24T00:34:32.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.849159+0000", 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4524e-05 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.852764+0000", 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036051989999999999 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.1b 54/53 [PushOp(3:dd4d3fbc:::rbd_data.115e8003c269.000000000000000b:head, version: 45'42, data_included: [0~89706,131072~131072,393216~131072,655360~131072], data_size: 482922, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:dd4d3fbc:::rbd_data.115e8003c269.000000000000000b:head@45'42, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8781824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.925189+0000", 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.4595606210000001, 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046247700000000003, 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.925189+0000", 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.925189+0000", 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.925190+0000", 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.925508+0000", 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031836100000000002 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.925535+0000", 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6361999999999999e-05 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.925557+0000", 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2670000000000001e-05 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.925653+0000", 2024-04-24T00:34:32.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5472999999999999e-05 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.925665+0000", 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1938e-05 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.929814+0000", 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041492179999999997 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(3.1b 54/53 [PushOp(3:daedd288:::rbd_data.115ee4742374.0000000000000003:head, version: 49'53, data_included: [0~167660,196608~131072], data_size: 298732, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:daedd288:::rbd_data.115ee4742374.0000000000000003:head@49'53, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.948845+0000", 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.4359044010000002, 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003045856, 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.948845+0000", 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.948845+0000", 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.948846+0000", 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2800000000000005e-07 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.949134+0000", 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028764699999999998 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.949163+0000", 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8544e-05 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.949182+0000", 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9250999999999999e-05 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.949265+0000", 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2917999999999995e-05 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.949273+0000", 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5769999999999994e-06 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.951891+0000", 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026179910000000001 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.482 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:32.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:32.484 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:32.479+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:32.585 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:32.579+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:32.652 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:32.652 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:32.652 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:32.652 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:32.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:32.685 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:32.683+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:32.740 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:32.740 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:32.741 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-24T00:34:32.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-04-24T00:34:32.786 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:32.784+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:32.794 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:32.794 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:32.794 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:32.794 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:32.794 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:32.816 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:32.887 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:32.883+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:32.988 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:32.988 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:32.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:32.988 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:32.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:5336 3.1d e36/21 3:b83bad9e:::rbd_data.115ed6c33904.0000000000000086:head v 36'3, mlcod=34'1)", 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.287185020999999, 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010615080000000001, 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694760+0000", 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8080000000000001e-06 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694891+0000", 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013035600000000001 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694916+0000", 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5392e-05 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694930+0000", 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4059999999999999e-05 2024-04-24T00:34:32.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695055+0000", 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124499 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695112+0000", 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7853000000000003e-05 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705273+0000", 2024-04-24T00:34:32.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010160626000000001 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705373+0000", 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100486 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10257 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:32.993 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.779541062, 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011690176, 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10257 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:32.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202404+0000", 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.897e-06 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202412+0000", 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202430+0000", 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8043e-05 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202442+0000", 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439999999999999e-05 2024-04-24T00:34:32.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207283+0000", 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048410600000000003 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207552+0000", 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026864100000000002 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.208755+0000", 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012028189999999999 2024-04-24T00:34:32.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.211687+0000", 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029325240000000002 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.213944+0000", 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022567300000000002 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214023+0000", 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8932999999999998e-05 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214093+0000", 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9538999999999999e-05 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10262 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:32.999 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.689197991, 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089239360000000004, 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10262 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292747+0000", 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.083e-06 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292753+0000", 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2700000000000001e-06 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292767+0000", 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4175999999999999e-05 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292776+0000", 2024-04-24T00:34:33.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0939999999999997e-06 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.295605+0000", 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028292909999999998 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.295896+0000", 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000290809 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296379+0000", 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048252899999999999 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.299342+0000", 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002962957 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301530+0000", 2024-04-24T00:34:33.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021886259999999999 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301605+0000", 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4893000000000003e-05 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301669+0000", 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4208000000000006e-05 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.005 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10263 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.689105775000002, 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091981449999999996, 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.006 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10263 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292839+0000", 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-06 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292846+0000", 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.81e-06 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292862+0000", 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6456e-05 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292870+0000", 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3699999999999995e-06 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296812+0000", 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039413900000000003 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296881+0000", 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9426999999999998e-05 2024-04-24T00:34:33.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.297166+0000", 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028513300000000002 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301682+0000", 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045159029999999999 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301927+0000", 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024475499999999998 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301980+0000", 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2796000000000001e-05 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.302036+0000", 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5992000000000002e-05 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.011 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10288 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.649197648000001, 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.009564744, 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10288 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.012 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332747+0000", 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:33.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332769+0000", 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1903e-05 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332780+0000", 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1243000000000001e-05 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332789+0000", 2024-04-24T00:34:33.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0699999999999996e-06 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336540+0000", 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037510339999999999 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336760+0000", 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021991099999999999 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.337418+0000", 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000658127 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.339101+0000", 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016826390000000001 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342150+0000", 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030494850000000002 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342236+0000", 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6212999999999996e-05 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342311+0000", 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4228999999999997e-05 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10359 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:33.017 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.538763243999998, 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091612789999999996, 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10359 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:33.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443182+0000", 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443189+0000", 2024-04-24T00:34:33.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1470000000000004e-06 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443206+0000", 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7603000000000001e-05 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443221+0000", 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4747e-05 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.448444+0000", 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052232700000000003 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.448605+0000", 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016098499999999999 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.449528+0000", 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092301199999999996 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451989+0000", 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024603250000000002 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452174+0000", 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018491100000000001 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452265+0000", 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1283e-05 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452341+0000", 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6656000000000006e-05 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.022 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10365 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.487171429, 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098439340000000004, 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10365 2024-04-24T00:34:33.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494773+0000", 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1399999999999995e-07 2024-04-24T00:34:33.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494782+0000", 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3649999999999993e-06 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494794+0000", 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1606e-05 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494804+0000", 2024-04-24T00:34:33.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0013999999999999e-05 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498700+0000", 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038961310000000002 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498969+0000", 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000269432 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.499771+0000", 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080119200000000005 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.503502+0000", 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037318300000000002 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504304+0000", 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080115000000000004 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504475+0000", 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017146199999999999 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504616+0000", 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014083800000000001 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10409 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:33.028 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.405589228, 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010506112, 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10409 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576355+0000", 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576359+0000", 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3229999999999999e-06 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576372+0000", 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576383+0000", 2024-04-24T00:34:33.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1192999999999999e-05 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580539+0000", 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041557230000000001 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580813+0000", 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027430400000000001 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.582037+0000", 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001223879 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.584201+0000", 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002164408 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586567+0000", 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023657090000000001 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586731+0000", 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016439800000000001 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586860+0000", 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128835 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10518 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.034 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.232516796999999, 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091784659999999997, 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10518 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:33.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2699999999999998e-07 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749432+0000", 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2780000000000004e-06 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749444+0000", 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1882000000000001e-05 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749451+0000", 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1890000000000001e-06 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.753811+0000", 2024-04-24T00:34:33.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043599609999999999 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.753978+0000", 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016679100000000001 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.754514+0000", 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00053602499999999998 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758097+0000", 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035836169999999999 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758429+0000", 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033133099999999998 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758527+0000", 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8919000000000006e-05 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758605+0000", 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7645999999999995e-05 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10645 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.954585145999999, 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010437706999999999, 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.040 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10645 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027359+0000", 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.06e-06 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027365+0000", 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2769999999999996e-06 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027380+0000", 2024-04-24T00:34:33.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5537e-05 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027389+0000", 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5299999999999996e-06 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.030838+0000", 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003449232 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031115+0000", 2024-04-24T00:34:33.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000277254 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031925+0000", 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080932599999999997 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.035618+0000", 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003693647 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037627+0000", 2024-04-24T00:34:33.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020091390000000001 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037720+0000", 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2775000000000005e-05 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037796+0000", 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.593e-05 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10664 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.918316896, 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088645240000000004, 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10664 2024-04-24T00:34:33.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:34:33.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063633+0000", 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5230000000000003e-06 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063645+0000", 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1725999999999999e-05 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063653+0000", 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6500000000000002e-06 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.068370+0000", 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047163049999999996 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.068639+0000", 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026901399999999998 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.069434+0000", 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00079544299999999995 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.071514+0000", 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020798180000000002 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072319+0000", 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080491299999999998 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072413+0000", 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4668999999999998e-05 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072491+0000", 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7589000000000005e-05 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10719 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.803413978999998, 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010945605000000001, 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.051 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10719 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:33.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178530+0000", 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1800000000000005e-07 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178535+0000", 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5129999999999998e-06 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178544+0000", 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2359999999999993e-06 2024-04-24T00:34:33.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178550+0000", 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.421e-06 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181685+0000", 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031346989999999999 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181965+0000", 2024-04-24T00:34:33.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027960399999999997 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.182619+0000", 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065424799999999996 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.186425+0000", 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038062690000000001 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189312+0000", 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028871809999999999 2024-04-24T00:34:33.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189407+0000", 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4345000000000006e-05 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189475+0000", 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8270999999999997e-05 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10739 3.10 3:0a996d09:::rbd_data.115e380d621c.0000000000000199:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.760517477, 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092681780000000002, 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10739 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221428+0000", 2024-04-24T00:34:33.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0949999999999998e-06 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221438+0000", 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9310000000000003e-06 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221463+0000", 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5023000000000001e-05 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221477+0000", 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3591000000000001e-05 2024-04-24T00:34:33.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221633+0000", 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156565 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.222026+0000", 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039234399999999998 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.223749+0000", 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017237109999999999 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230482+0000", 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067323670000000004 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230517+0000", 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981999999999998e-05 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230610+0000", 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230694+0000", 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4393000000000003e-05 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10768 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.699286572999998, 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092323920000000007, 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.062 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10768 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282658+0000", 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1799999999999999e-06 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282664+0000", 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6339999999999998e-06 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282676+0000", 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2320000000000001e-05 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282687+0000", 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1103e-05 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284651+0000", 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00196408 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284827+0000", 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175768 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.285720+0000", 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089352799999999999 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289504+0000", 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003783813 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291726+0000", 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022220339999999999 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291814+0000", 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7425999999999999e-05 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291889+0000", 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5506000000000005e-05 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10769 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.699201995999999, 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012818598, 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10769 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.068 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282742+0000", 2024-04-24T00:34:33.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282747+0000", 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0329999999999999e-06 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282759+0000", 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1687e-05 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282768+0000", 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.286885+0000", 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041171139999999998 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.287035+0000", 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149968 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.288104+0000", 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010690540000000001 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295331+0000", 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072273800000000003 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295418+0000", 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295493+0000", 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4825000000000003e-05 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295560+0000", 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7191000000000001e-05 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10770 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:33.073 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.699162746999999, 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013929258999999999, 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10770 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:33.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282782+0000", 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9000000000000005e-07 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282786+0000", 2024-04-24T00:34:33.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1330000000000001e-06 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282795+0000", 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-06 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282803+0000", 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8499999999999994e-06 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289625+0000", 2024-04-24T00:34:33.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068213689999999999 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289770+0000", 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014539899999999999 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.290476+0000", 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070591399999999998 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295352+0000", 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048766089999999996 2024-04-24T00:34:33.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296546+0000", 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011937580000000001 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296638+0000", 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2179e-05 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296710+0000", 2024-04-24T00:34:33.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1527999999999997e-05 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.699129646999999, 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.079 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:34:33.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:34:33.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:34:33.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:34:33.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:34:33.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.699093634, 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:34:33.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:34:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.44968924, 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:34:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:34:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.415652186999999, 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:34:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:34:33.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.101 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.102 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:33.102 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:32.984+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:33.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:33.104 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:34:33.079+0000 7f841a5f8640 -1 mon.a@0(leader).osd e58 definitely_dead 0 2024-04-24T00:34:33.147 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:33.144+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:33.206 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:33.200+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:33.273 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:33.273 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:33.273 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:33.273 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:33.290 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:33.307 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:33.303+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:33.407 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:33.404+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:33.454 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:33.454 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:33.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:33.508 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:33.503+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:33.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:33.609 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:33.604+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.710093860999997, 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.642 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:33.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:34:33.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:33.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:33.648 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.255865127, 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024520601, 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 117 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:33.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380635+0000", 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380642+0000", 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7700000000000004e-06 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380654+0000", 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1686e-05 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380666+0000", 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2150000000000001e-05 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399787+0000", 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019121461999999999 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399921+0000", 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134043 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400127+0000", 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020607500000000001 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401186+0000", 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001058738 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.404952+0000", 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037661779999999998 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405042+0000", 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9889999999999995e-05 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405155+0000", 2024-04-24T00:34:33.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112812 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4984 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.042749914000002, 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024665092, 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.654 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4984 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4e-07 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593755+0000", 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6829999999999999e-06 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593765+0000", 2024-04-24T00:34:33.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7140000000000005e-06 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593773+0000", 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7829999999999999e-06 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609097+0000", 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015324669000000001 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609254+0000", 2024-04-24T00:34:33.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015632499999999999 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609918+0000", 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066479400000000002 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.615678+0000", 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057599119999999998 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618123+0000", 2024-04-24T00:34:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024450909999999999 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618280+0000", 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156706 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618415+0000", 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134475 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5029 3.f 3:f41ea3ee:::rbd_data.115ed6c33904.0000000000000006:head [zero 262144~262144,zero 786432~65536,zero 917504~39917,write 957421~8409 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.027298549000001, 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024904274000000001, 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5029 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609208+0000", 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8959999999999997e-06 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609285+0000", 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6738999999999998e-05 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609308+0000", 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3896999999999998e-05 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609323+0000", 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4543e-05 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622194+0000", 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012870985 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622559+0000", 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036503000000000002 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.623244+0000", 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068464100000000005 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.629548+0000", 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063044329999999999 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.633948+0000", 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044003250000000001 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634037+0000", 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9109000000000004e-05 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634105+0000", 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7675999999999994e-05 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5037 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.610513+0000", 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.025986261, 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024006709000000001, 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5037 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610513+0000", 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610513+0000", 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610516+0000", 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7099999999999999e-06 2024-04-24T00:34:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610530+0000", 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3956e-05 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610560+0000", 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0236999999999999e-05 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610581+0000", 2024-04-24T00:34:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0622999999999999e-05 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.624806+0000", 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014225193000000001 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.625000+0000", 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019424500000000001 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.625799+0000", 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00079852000000000005 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.633823+0000", 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080246020000000005 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634233+0000", 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000410169 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634405+0000", 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017113499999999999 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634520+0000", 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115319 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5305 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.685285+0000", 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.95121477, 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023678181999999999, 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5305 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685285+0000", 2024-04-24T00:34:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685285+0000", 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685286+0000", 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-06 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685293+0000", 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7270000000000001e-06 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685308+0000", 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5216e-05 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685317+0000", 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6299999999999993e-06 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.693625+0000", 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083071580000000003 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694495+0000", 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00087003500000000001 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695368+0000", 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00087324600000000005 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.699580+0000", 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042119280000000002 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.708785+0000", 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092054130000000008 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.708871+0000", 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5377000000000006e-05 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.708963+0000", 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2281999999999995e-05 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5306 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.951162950000001, 2024-04-24T00:34:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025510631999999998, 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5306 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685337+0000", 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685342+0000", 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4560000000000002e-06 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685352+0000", 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840000000000006e-06 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685359+0000", 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7100000000000007e-06 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696262+0000", 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010902604 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696476+0000", 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021439300000000001 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.697460+0000", 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00098361900000000003 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.706972+0000", 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095118609999999999 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710681+0000", 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037093619999999999 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710762+0000", 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0374e-05 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710847+0000", 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5478999999999999e-05 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5317 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.687320+0000", 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.949179151999999, 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023672143, 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5317 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.687320+0000", 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.687320+0000", 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.687322+0000", 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.81e-06 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.687331+0000", 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5199999999999997e-06 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.687352+0000", 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1056999999999998e-05 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.687368+0000", 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.624e-05 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.699673+0000", 2024-04-24T00:34:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012305482 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.699865+0000", 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019141500000000001 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.700767+0000", 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00090185199999999997 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.707069+0000", 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063021249999999996 2024-04-24T00:34:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710858+0000", 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037890630000000001 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710919+0000", 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1032000000000001e-05 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710992+0000", 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3547000000000006e-05 2024-04-24T00:34:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10360 3.11 3:8fda792a:::rbd_data.115e380d621c.0000000000000177:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.185108981999999, 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042632046999999999, 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10360 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451393+0000", 2024-04-24T00:34:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5600000000000001e-06 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451406+0000", 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2984e-05 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451445+0000", 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9045999999999999e-05 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451461+0000", 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5786000000000001e-05 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451608+0000", 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014694900000000001 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451914+0000", 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306657 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002601744 2024-04-24T00:34:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493753+0000", 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039236409 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493841+0000", 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8448000000000004e-05 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493931+0000", 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0198999999999995e-05 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494023+0000", 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1265000000000001e-05 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10360 3.11 e40/21 ondisk, result = 0)", 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.174243689000001, 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031780810999999999, 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462258+0000", 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2539999999999999e-06 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462266+0000", 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8029999999999997e-06 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462294+0000", 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8637e-05 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462314+0000", 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493814+0000", 2024-04-24T00:34:33.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031500133 2024-04-24T00:34:33.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493831+0000", 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7357e-05 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494037+0000", 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020520400000000001 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.698 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.778128172999999, 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:34:33.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:34:33.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.703 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.776879425000001, 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:34:33.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10590 3.13 e41/21 ondisk, result = 0)", 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.765633841, 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042592342999999998, 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:33.709 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870868+0000", 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1730000000000002e-06 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870876+0000", 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2069999999999995e-06 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870900+0000", 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.404e-05 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870914+0000", 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.366e-05 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913167+0000", 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042252855999999998 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913184+0000", 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7779e-05 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913458+0000", 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027362799999999999 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10592 3.16 e41/21 ondisk, result = 0)", 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.761217245000001, 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044174204000000002, 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875284+0000", 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1050000000000002e-06 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875292+0000", 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3200000000000002e-06 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875319+0000", 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7382999999999999e-05 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875334+0000", 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4786e-05 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919179+0000", 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043845151999999998 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919195+0000", 2024-04-24T00:34:33.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5846999999999999e-05 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919456+0000", 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026161100000000003 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.619255321000001, 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.716 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:33.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.721 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.605760778, 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21553 3.7 3:e2dbd82a:::rbd_data.115eeb9272a5.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.5402776730000003, 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029891540000000001, 2024-04-24T00:34:33.725 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21553 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096223+0000", 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3570000000000001e-06 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096230+0000", 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4080000000000003e-06 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096244+0000", 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4236999999999999e-05 2024-04-24T00:34:33.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096251+0000", 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6569999999999997e-06 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096335+0000", 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3997000000000002e-05 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096473+0000", 2024-04-24T00:34:33.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013718199999999999 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.097939+0000", 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014665570000000001 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125952+0000", 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028013198 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125977+0000", 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4613999999999998e-05 2024-04-24T00:34:33.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126035+0000", 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7966e-05 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126113+0000", 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8367000000000004e-05 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.730 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21553 3.7 e52/44 ondisk, result = 0)", 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.102601+0000", 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.533898669, 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023360999, 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.102601+0000", 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.102601+0000", 2024-04-24T00:34:33.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.102602+0000", 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3510000000000001e-06 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.102608+0000", 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6620000000000002e-06 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.102624+0000", 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6707000000000001e-05 2024-04-24T00:34:33.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.102633+0000", 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2910000000000007e-06 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125933+0000", 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023300465999999999 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125946+0000", 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2645e-05 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125962+0000", 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5877000000000001e-05 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.3439796629999998, 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:33.734 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21613 3.7 e52/44 ondisk, result = 0)", 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.3361457239999996, 2024-04-24T00:34:33.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044307553999999999, 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300355+0000", 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4610000000000001e-06 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300362+0000", 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4930000000000003e-06 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300382+0000", 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9887000000000001e-05 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300394+0000", 2024-04-24T00:34:33.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2799000000000001e-05 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344558+0000", 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044164017999999999 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344589+0000", 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.027e-05 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:33.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344661+0000", 2024-04-24T00:34:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2626000000000006e-05 2024-04-24T00:34:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:33.743 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:33.743 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:33.744 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:33.744 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:33.745 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:33.745 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:33.745 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:33.740+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:33.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:33.748 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:33.748 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:33.765 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:33.765 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:33.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:34:33.777 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.0. 2024-04-24T00:34:33.798 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:33.798 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:33.822 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:33.822 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:33.843 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:33.843 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:33.846 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:33.843+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:33.864 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:33.864 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:33.875 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:33.875 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:33.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:33.982 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:33.979+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:34.048 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:34.043+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:34.050 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:34.050 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:34.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:34.149 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:34.144+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:34.232 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:34.232 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:34.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:34.250 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:34.244+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:34.351 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:34.347+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:34.370 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:34.370 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:34.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:34.469 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:34.467+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:34.535 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:34:34.546 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:34.546 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:34.552 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:34.547+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:34.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:34.653 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:34.648+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:34.735 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:34.735 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:34.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:34.793 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:34.791+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:34.855 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:34.851+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:34.868 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:34.868 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:34.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:34.956 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:34.952+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:35.050 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:35.050 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:35.057 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:35.052+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:35.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:35.158 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:35.152+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:35.186 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:35.186 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:35.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:35.259 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:35.255+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:35.327 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:35.327 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:35.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:35.385 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:35.383+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:35.461 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:35.455+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:35.511 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:35.511 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:35.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:35.569 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:35.567+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:35.663 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:35.660+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:35.691 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:35.691 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:35.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:35.764 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:35.759+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:35.837 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:35.837 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:35.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:35.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:35.899 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:35.896+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:35.965 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:35.963+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:35.967 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:35.967 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:35.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:36.042 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:36.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:36.059 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:36.067 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:36.064+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:36.075 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:36.075 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:36.091 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:36.092 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:36.145 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:36.145 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:36.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:36.167 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:36.163+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:36.268 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:36.264+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:36.308 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-04-24T00:34:36.315 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:36.315 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:36.333 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:36.333 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:36.344 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:36.344 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:36.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:36.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:36.374 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:36.391 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:36.388+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:36.402 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:36.403 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:36.470 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:36.464+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:36.513 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:36.513 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:36.513 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:36.513 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:36.513 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:36.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:36.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:34:36.567 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:36.567 INFO:teuthology.orchestra.run.smithi064.stdout:1228 read 0x64481d1 thru 0x6452cf3 (0xab23 bytes) 2024-04-24T00:34:36.570 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:36.568+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:36.571 INFO:teuthology.orchestra.run.smithi064.stdout:1229 read 0x695bc47 thru 0x696b8c1 (0xfc7b bytes) 2024-04-24T00:34:36.572 INFO:teuthology.orchestra.run.smithi064.stdout:1230 read 0x40fae40 thru 0x410652d (0xb6ee bytes) 2024-04-24T00:34:36.574 INFO:teuthology.orchestra.run.smithi064.stdout:1231 write 0x4c3e12e thru 0x4c420b0 (0x3f83 bytes) 2024-04-24T00:34:36.671 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:36.667+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:36.698 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:36.698 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:36.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:36.698 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:36.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.698 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:34:36.698 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.9845212239999999, 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:34:36.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:34:36.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.9844232989999995, 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:36.704 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:36.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.710 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21662 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.9046728970000002, 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082640219999999993, 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21662 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788217+0000", 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788221+0000", 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0470000000000004e-06 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788234+0000", 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3138e-05 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788243+0000", 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1759999999999993e-06 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791802+0000", 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003559347 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791936+0000", 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134349 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.796480+0000", 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004544022 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.714 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21667 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.801921+0000", 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.8909684450000004, 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062889759999999999, 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.715 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21667 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801921+0000", 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801921+0000", 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801923+0000", 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3e-06 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801935+0000", 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2068e-05 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801958+0000", 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3124000000000001e-05 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.801972+0000", 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3485e-05 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.802045+0000", 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2924999999999997e-05 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.802151+0000", 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106289 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.802330+0000", 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017869699999999999 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.804359+0000", 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020295410000000002 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.807858+0000", 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034990049999999999 2024-04-24T00:34:36.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.807958+0000", 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010035 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.808210+0000", 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025119200000000002 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22086 3.1c 3:3fa7153c:::rbd_data.115eeb9272a5.0000000000000002:11 [sparse-read 524288~524288 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.720 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.794213+0000", 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.898676247, 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062811129999999996, 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22086 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794213+0000", 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794213+0000", 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794214+0000", 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9199999999999999e-07 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794217+0000", 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6229999999999999e-06 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794228+0000", 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0933e-05 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794234+0000", 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0360000000000003e-06 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.796826+0000", 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025921550000000001 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.797065+0000", 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023870200000000001 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.800494+0000", 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034287720000000001 2024-04-24T00:34:36.724 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22089 3.1c 3:3fa7153c:::rbd_data.115eeb9272a5.0000000000000002:11 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.794311+0000", 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8985781140000002, 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064035760000000002, 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.725 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22089 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794311+0000", 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794311+0000", 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794312+0000", 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9500000000000001e-07 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794318+0000", 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5250000000000002e-06 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794338+0000", 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9871e-05 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794345+0000", 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8050000000000001e-06 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.800534+0000", 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061887909999999999 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.800618+0000", 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4291999999999999e-05 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.800715+0000", 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6496999999999994e-05 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22095 3.1c 3:3fa7153c:::rbd_data.115eeb9272a5.0000000000000002:11 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.794499+0000", 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8983900440000001, 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070792279999999999, 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22095 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794499+0000", 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794499+0000", 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794500+0000", 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2799999999999995e-07 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794503+0000", 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5750000000000001e-06 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794515+0000", 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794520+0000", 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6300000000000003e-06 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.800825+0000", 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063040769999999999 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.801118+0000", 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029312099999999997 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.801578+0000", 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00046063300000000001 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22145 3.1d 3:bf5208ff:::rbd_data.115e6a380959.0000000000000001:head [write 0~524288 [fadvise_sequential+fadvise_nocache],zero 1048576~311524,write 1360100~9561 [fadvise_sequential+fadvise_nocache],zero 2621440~524288,zero 6815744~1717,write 6817461~5360 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.804836+0000", 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8880527150000002, 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015668827, 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.733 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22145 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.804836+0000", 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.804836+0000", 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.804837+0000", 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6899999999999996e-07 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805641+0000", 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080407199999999997 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805653+0000", 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1240000000000001e-05 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805660+0000", 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5020000000000003e-06 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805736+0000", 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5724999999999997e-05 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805889+0000", 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015369299999999999 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.806194+0000", 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030478699999999999 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820040+0000", 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013846089000000001 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820279+0000", 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023887499999999999 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820435+0000", 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015621100000000001 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820505+0000", 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9764000000000005e-05 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.738 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22149 3.c 3:3770836c:::rbd_data.115e6a380959.0000000000000006:head [write 0~524288 [fadvise_sequential+fadvise_nocache],zero 524288~524288,zero 1572864~524288,zero 5767168~1048576,zero 8388608~422408,write 8811016~8118 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.810003+0000", 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.882885881, 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01478607, 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22149 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810003+0000", 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810003+0000", 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810005+0000", 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2050000000000001e-06 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810532+0000", 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00052670099999999997 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810553+0000", 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1285999999999999e-05 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810566+0000", 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2551000000000001e-05 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810682+0000", 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116123 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810975+0000", 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029288900000000001 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811371+0000", 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039598999999999998 2024-04-24T00:34:36.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824401+0000", 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013030168 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824615+0000", 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021383400000000001 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824706+0000", 2024-04-24T00:34:36.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1043000000000002e-05 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824789+0000", 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3280000000000002e-05 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22150 3.c 3:36ffc5a2:::rbd_data.115e6a380959.0000000000000003:head [write 0~262144 [fadvise_sequential+fadvise_nocache],zero 262144~196158,write 458302~37038 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.811490+0000", 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8813994579999997, 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016278401000000001, 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.744 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22150 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811490+0000", 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811490+0000", 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811491+0000", 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1119999999999999e-06 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811826+0000", 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000335547 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811837+0000", 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0727000000000001e-05 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811848+0000", 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1355000000000001e-05 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820484+0000", 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086351369999999993 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820601+0000", 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011789400000000001 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820805+0000", 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020369799999999999 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825543+0000", 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047380230000000001 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:36.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.827715+0000", 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002172186 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.827746+0000", 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0229000000000001e-05 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.827768+0000", 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2493e-05 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22152 3.1c 3:3c0f4810:::rbd_data.115e6a380959.0000000000000008:head [zero 0~103219,write 103219~11656 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.813060+0000", 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8798287460000003, 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013815094, 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22152 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.750 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813060+0000", 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813060+0000", 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813061+0000", 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813078+0000", 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7125e-05 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813089+0000", 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0794e-05 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813114+0000", 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4272000000000001e-05 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824337+0000", 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011223568 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824486+0000", 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148942 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824613+0000", 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012721400000000001 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826048+0000", 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001434301 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826808+0000", 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00076005199999999999 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826849+0000", 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1198000000000001e-05 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826875+0000", 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6625e-05 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22145 3.1d e58/53 ondisk, result = 0)", 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.813261+0000", 2024-04-24T00:34:36.755 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8796282500000001, 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068002590000000003, 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813261+0000", 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813261+0000", 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813262+0000", 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.48e-06 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813271+0000", 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7649999999999999e-06 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813294+0000", 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2654999999999999e-05 2024-04-24T00:34:36.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813310+0000", 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6599000000000002e-05 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819994+0000", 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066839389999999999 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820013+0000", 2024-04-24T00:34:36.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8597000000000002e-05 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820061+0000", 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8223999999999998e-05 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22164 3.c 3:30fd3d95:::rbd_data.115eeb9272a5.000000000000000d:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.817271+0000", 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8756180220000003, 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075233560000000001, 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.759 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22164 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817271+0000", 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817271+0000", 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817272+0000", 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6600000000000005e-07 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817276+0000", 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6720000000000002e-06 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817283+0000", 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6879999999999999e-06 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817289+0000", 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6339999999999998e-06 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824455+0000", 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0071660409999999997 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824794+0000", 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033945500000000001 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22168 3.1c 3:381c9e4b:::rbd_data.115eeb9272a5.0000000000000015:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.817343+0000", 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8755457419999999, 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082038319999999994, 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.763 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22168 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817343+0000", 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817343+0000", 2024-04-24T00:34:36.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817344+0000", 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817348+0000", 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8929999999999998e-06 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817355+0000", 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6610000000000003e-06 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817360+0000", 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6080000000000003e-06 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825264+0000", 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079035140000000004 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825547+0000", 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028333299999999998 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22182 3.c 3:3159bf60:::rbd_data.115eeb9272a5.000000000000002c:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.817414+0000", 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.875475507, 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007932616, 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:36.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22182 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817414+0000", 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817414+0000", 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817414+0000", 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5499999999999998e-07 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817416+0000", 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817422+0000", 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0859999999999998e-06 2024-04-24T00:34:36.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817428+0000", 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2639999999999997e-06 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825222+0000", 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0077935230000000001 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825346+0000", 2024-04-24T00:34:36.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124265 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22188 3.c 3:3391bdb6:::rbd_data.115eeb9272a5.0000000000000037:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.817481+0000", 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8754080869999998, 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080402930000000004, 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.771 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22188 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817481+0000", 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817481+0000", 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817481+0000", 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5499999999999998e-07 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817484+0000", 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2630000000000002e-06 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817494+0000", 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0689999999999999e-05 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817500+0000", 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1050000000000003e-06 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825401+0000", 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079010790000000001 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825521+0000", 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120701 2024-04-24T00:34:36.774 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22190 3.1c 3:3fa7153c:::rbd_data.115eeb9272a5.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.818453+0000", 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8744358779999999, 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072144100000000001, 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.775 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22190 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818453+0000", 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818453+0000", 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818454+0000", 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7499999999999999e-07 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818458+0000", 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3880000000000002e-06 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818472+0000", 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.398e-05 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818480+0000", 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2729999999999997e-06 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825548+0000", 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070680250000000003 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825605+0000", 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6907999999999998e-05 2024-04-24T00:34:36.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825668+0000", 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2960999999999996e-05 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22213 3.c 3:31523c56:::rbd_data.115eeb9272a5.000000000000002b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.819091+0000", 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8737983279999999, 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066602509999999998, 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.779 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22213 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819091+0000", 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819091+0000", 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819091+0000", 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.89e-07 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819094+0000", 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4729999999999999e-06 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819100+0000", 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.516e-06 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819107+0000", 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8000000000000001e-06 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825590+0000", 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064826490000000001 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825751+0000", 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016132399999999999 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22214 3.c 3:3159bf60:::rbd_data.115eeb9272a5.000000000000002c:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.819115+0000", 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.8737740060000001, 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067134639999999997, 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.783 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22214 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819115+0000", 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819115+0000", 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819116+0000", 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2300000000000002e-07 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819118+0000", 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1009999999999999e-06 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819124+0000", 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0020000000000003e-06 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819129+0000", 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7880000000000003e-06 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825767+0000", 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066375469999999997 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825829+0000", 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1603e-05 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.786 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22220 3.c 3:3391bdb6:::rbd_data.115eeb9272a5.0000000000000037:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.819158+0000", 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.873731576, 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068919289999999998, 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22220 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.787 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819158+0000", 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819158+0000", 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819158+0000", 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819160+0000", 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.057e-06 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819166+0000", 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6060000000000002e-06 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.819170+0000", 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4170000000000002e-06 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825885+0000", 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067145169999999997 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826049+0000", 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016490799999999999 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:36.790 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:36.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:36.792 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:36.788+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:36.893 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:36.891+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:36.960 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:36.961 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:36.961 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:36.961 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:36.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:36.994 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:36.990+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:37.095 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:37.092+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:37.098 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:37.098 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:37.098 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:37.098 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:37.098 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:37.112 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:37.199 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:37.196+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:37.282 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.282 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:5336 3.1d e36/21 3:b83bad9e:::rbd_data.115ed6c33904.0000000000000086:head v 36'3, mlcod=34'1)", 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.581581429, 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010615080000000001, 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694760+0000", 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8080000000000001e-06 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694891+0000", 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013035600000000001 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694916+0000", 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5392e-05 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694930+0000", 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4059999999999999e-05 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695055+0000", 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124499 2024-04-24T00:34:37.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695112+0000", 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7853000000000003e-05 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705273+0000", 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010160626000000001 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705373+0000", 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100486 2024-04-24T00:34:37.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10257 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.073937470000001, 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011690176, 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10257 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202404+0000", 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.897e-06 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202412+0000", 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202430+0000", 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8043e-05 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202442+0000", 2024-04-24T00:34:37.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439999999999999e-05 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207283+0000", 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048410600000000003 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207552+0000", 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026864100000000002 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.208755+0000", 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012028189999999999 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.211687+0000", 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029325240000000002 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.213944+0000", 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022567300000000002 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214023+0000", 2024-04-24T00:34:37.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8932999999999998e-05 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214093+0000", 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9538999999999999e-05 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10262 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.983594399000001, 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089239360000000004, 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.292 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10262 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292745+0000", 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292747+0000", 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.083e-06 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292753+0000", 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2700000000000001e-06 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292767+0000", 2024-04-24T00:34:37.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4175999999999999e-05 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292776+0000", 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0939999999999997e-06 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.295605+0000", 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028292909999999998 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.295896+0000", 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000290809 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296379+0000", 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048252899999999999 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.299342+0000", 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002962957 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301530+0000", 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021886259999999999 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301605+0000", 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4893000000000003e-05 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301669+0000", 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4208000000000006e-05 2024-04-24T00:34:37.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10263 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.983502182999999, 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091981449999999996, 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10263 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292839+0000", 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-06 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292846+0000", 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.81e-06 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292862+0000", 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6456e-05 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292870+0000", 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3699999999999995e-06 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296812+0000", 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039413900000000003 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296881+0000", 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9426999999999998e-05 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.297166+0000", 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028513300000000002 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301682+0000", 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045159029999999999 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301927+0000", 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024475499999999998 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301980+0000", 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2796000000000001e-05 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.302036+0000", 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5992000000000002e-05 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10288 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.943594055999998, 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.009564744, 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10288 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332747+0000", 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332769+0000", 2024-04-24T00:34:37.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1903e-05 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332780+0000", 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1243000000000001e-05 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332789+0000", 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0699999999999996e-06 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336540+0000", 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037510339999999999 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336760+0000", 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021991099999999999 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.337418+0000", 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000658127 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.339101+0000", 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016826390000000001 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342150+0000", 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030494850000000002 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342236+0000", 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6212999999999996e-05 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342311+0000", 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4228999999999997e-05 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10359 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.833159651999999, 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091612789999999996, 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.309 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10359 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443182+0000", 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443189+0000", 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1470000000000004e-06 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443206+0000", 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7603000000000001e-05 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443221+0000", 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4747e-05 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.448444+0000", 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052232700000000003 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.448605+0000", 2024-04-24T00:34:37.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016098499999999999 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.449528+0000", 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092301199999999996 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451989+0000", 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024603250000000002 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452174+0000", 2024-04-24T00:34:37.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018491100000000001 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452265+0000", 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1283e-05 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452341+0000", 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6656000000000006e-05 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10365 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.781567837000001, 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098439340000000004, 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10365 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494773+0000", 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1399999999999995e-07 2024-04-24T00:34:37.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494782+0000", 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3649999999999993e-06 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494794+0000", 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1606e-05 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494804+0000", 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0013999999999999e-05 2024-04-24T00:34:37.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498700+0000", 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038961310000000002 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498969+0000", 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000269432 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.499771+0000", 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080119200000000005 2024-04-24T00:34:37.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.503502+0000", 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037318300000000002 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504304+0000", 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080115000000000004 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504475+0000", 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017146199999999999 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504616+0000", 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014083800000000001 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10409 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.699985636000001, 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010506112, 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.320 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10409 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576355+0000", 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576359+0000", 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3229999999999999e-06 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576372+0000", 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576383+0000", 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1192999999999999e-05 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580539+0000", 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041557230000000001 2024-04-24T00:34:37.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580813+0000", 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027430400000000001 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.582037+0000", 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001223879 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.584201+0000", 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002164408 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586567+0000", 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023657090000000001 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586731+0000", 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016439800000000001 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586860+0000", 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128835 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.325 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10518 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.526913205, 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091784659999999997, 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10518 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2699999999999998e-07 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749432+0000", 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2780000000000004e-06 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749444+0000", 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1882000000000001e-05 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749451+0000", 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1890000000000001e-06 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.753811+0000", 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043599609999999999 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.753978+0000", 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016679100000000001 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.754514+0000", 2024-04-24T00:34:37.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00053602499999999998 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758097+0000", 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035836169999999999 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758429+0000", 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033133099999999998 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758527+0000", 2024-04-24T00:34:37.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8919000000000006e-05 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758605+0000", 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7645999999999995e-05 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10645 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.248981554, 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010437706999999999, 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.331 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10645 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027359+0000", 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.06e-06 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027365+0000", 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2769999999999996e-06 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027380+0000", 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5537e-05 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027389+0000", 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5299999999999996e-06 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.030838+0000", 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003449232 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031115+0000", 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000277254 2024-04-24T00:34:37.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031925+0000", 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080932599999999997 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.035618+0000", 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003693647 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037627+0000", 2024-04-24T00:34:37.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020091390000000001 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037720+0000", 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2775000000000005e-05 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037796+0000", 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.593e-05 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.336 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10664 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.212713304000001, 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088645240000000004, 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10664 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063627+0000", 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063633+0000", 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5230000000000003e-06 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063645+0000", 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1725999999999999e-05 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.063653+0000", 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6500000000000002e-06 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.068370+0000", 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047163049999999996 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.068639+0000", 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026901399999999998 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.069434+0000", 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00079544299999999995 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.071514+0000", 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020798180000000002 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072319+0000", 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080491299999999998 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072413+0000", 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4668999999999998e-05 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.072491+0000", 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7589000000000005e-05 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10719 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.097810386999999, 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010945605000000001, 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.342 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10719 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178530+0000", 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1800000000000005e-07 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178535+0000", 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5129999999999998e-06 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178544+0000", 2024-04-24T00:34:37.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2359999999999993e-06 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178550+0000", 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.421e-06 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181685+0000", 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031346989999999999 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181965+0000", 2024-04-24T00:34:37.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027960399999999997 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.182619+0000", 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065424799999999996 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.186425+0000", 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038062690000000001 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189312+0000", 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028871809999999999 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189407+0000", 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4345000000000006e-05 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189475+0000", 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8270999999999997e-05 2024-04-24T00:34:37.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10739 3.10 3:0a996d09:::rbd_data.115e380d621c.0000000000000199:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.054913885000001, 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092681780000000002, 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.348 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10739 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221428+0000", 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0949999999999998e-06 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221438+0000", 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9310000000000003e-06 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221463+0000", 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5023000000000001e-05 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221477+0000", 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3591000000000001e-05 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221633+0000", 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156565 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.222026+0000", 2024-04-24T00:34:37.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039234399999999998 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.223749+0000", 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017237109999999999 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230482+0000", 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067323670000000004 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230517+0000", 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981999999999998e-05 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230610+0000", 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230694+0000", 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4393000000000003e-05 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10768 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.993682980999999, 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092323920000000007, 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10768 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.354 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282658+0000", 2024-04-24T00:34:37.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1799999999999999e-06 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282664+0000", 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6339999999999998e-06 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282676+0000", 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2320000000000001e-05 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282687+0000", 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1103e-05 2024-04-24T00:34:37.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284651+0000", 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00196408 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284827+0000", 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175768 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.285720+0000", 2024-04-24T00:34:37.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089352799999999999 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289504+0000", 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003783813 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291726+0000", 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022220339999999999 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291814+0000", 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7425999999999999e-05 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291889+0000", 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5506000000000005e-05 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10769 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:37.359 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.993598404, 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012818598, 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10769 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282742+0000", 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282747+0000", 2024-04-24T00:34:37.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0329999999999999e-06 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282759+0000", 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1687e-05 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282768+0000", 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.286885+0000", 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041171139999999998 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.287035+0000", 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149968 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.288104+0000", 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010690540000000001 2024-04-24T00:34:37.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295331+0000", 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072273800000000003 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295418+0000", 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295493+0000", 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4825000000000003e-05 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295560+0000", 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7191000000000001e-05 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.365 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10770 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.993559155, 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013929258999999999, 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10770 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.366 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282782+0000", 2024-04-24T00:34:37.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9000000000000005e-07 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282786+0000", 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1330000000000001e-06 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282795+0000", 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-06 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282803+0000", 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8499999999999994e-06 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289625+0000", 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068213689999999999 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289770+0000", 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014539899999999999 2024-04-24T00:34:37.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.290476+0000", 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070591399999999998 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295352+0000", 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048766089999999996 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296546+0000", 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011937580000000001 2024-04-24T00:34:37.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296638+0000", 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2179e-05 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296710+0000", 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1527999999999997e-05 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.993526055, 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:37.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:34:37.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:34:37.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.993490042000001, 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.377 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:34:37.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:34:37.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:34:37.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.744085647999999, 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:34:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.710048595, 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.393 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:37.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:37.395 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:37.390+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:37.496 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:37.492+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:37.557 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:34:37.569 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:37.570 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:37.570 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:37.570 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:37.586 INFO:teuthology.orchestra.run.smithi064.stdout:1232 read 0x5eaeed8 thru 0x5eb6be0 (0x7d09 bytes) 2024-04-24T00:34:37.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:37.601 INFO:teuthology.orchestra.run.smithi064.stdout:1233 write 0xab93cc2 thru 0xab9abe1 (0x6f20 bytes) 2024-04-24T00:34:37.601 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:37.598+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:37.613 INFO:teuthology.orchestra.run.smithi064.stdout:1234 write 0xb477274 thru 0xb47ec23 (0x79b0 bytes) 2024-04-24T00:34:37.617 INFO:teuthology.orchestra.run.smithi064.stdout:1237 read 0x10d15e5 thru 0x10d48bc (0x32d8 bytes) 2024-04-24T00:34:37.618 INFO:teuthology.orchestra.run.smithi064.stdout:1238 writesame 0xa79984c thru 0xa79ac54 data_size 0x1(0x1409 bytes) 2024-04-24T00:34:37.626 INFO:teuthology.orchestra.run.smithi064.stdout:1240 writesame 0x3534ec2 thru 0x354172a data_size 0x1f(0xc869 bytes) 2024-04-24T00:34:37.632 INFO:teuthology.orchestra.run.smithi064.stdout:1241 punch from 0x7cfde17 to 0x7d071b4, (0x939d bytes) 2024-04-24T00:34:37.647 INFO:teuthology.orchestra.run.smithi064.stdout:1242 trunc from 0xb47ec24 to 0x27275ff 2024-04-24T00:34:37.690 INFO:teuthology.orchestra.run.smithi064.stdout:1243 write 0xc3da7c2 thru 0xc3e2679 (0x7eb8 bytes) 2024-04-24T00:34:37.692 INFO:teuthology.orchestra.run.smithi064.stdout:1244 read 0xfd4fe thru 0x10d000 (0xfb03 bytes) 2024-04-24T00:34:37.693 INFO:teuthology.orchestra.run.smithi064.stdout:1246 write 0xac0705d thru 0xac0c747 (0x56eb bytes) 2024-04-24T00:34:37.698 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:37.694+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:37.709 INFO:teuthology.orchestra.run.smithi064.stdout:1247 trunc from 0xc3e267a to 0x6a2a52c 2024-04-24T00:34:37.728 INFO:teuthology.orchestra.run.smithi064.stdout:1249 read 0x5d31fac thru 0x5d324ac (0x501 bytes) 2024-04-24T00:34:37.729 INFO:teuthology.orchestra.run.smithi064.stdout:1250 punch from 0x4e6716e to 0x4e74565, (0xd3f7 bytes) 2024-04-24T00:34:37.729 INFO:teuthology.orchestra.run.smithi064.stdout:1251 read 0x408ca06 thru 0x408f9b3 (0x2fae bytes) 2024-04-24T00:34:37.751 INFO:teuthology.orchestra.run.smithi064.stdout:1252 write 0xa3b9ee8 thru 0xa3c11b3 (0x72cc bytes) 2024-04-24T00:34:37.755 INFO:teuthology.orchestra.run.smithi064.stdout:1253 read 0x8ce4acb thru 0x8cebca2 (0x71d8 bytes) 2024-04-24T00:34:37.756 INFO:teuthology.orchestra.run.smithi064.stdout:1255 write 0x94da1f0 thru 0x94db8ec (0x16fd bytes) 2024-04-24T00:34:37.760 INFO:teuthology.orchestra.run.smithi064.stdout:1257 punch from 0x2cfe956 to 0x2d0de58, (0xf502 bytes) 2024-04-24T00:34:37.760 INFO:teuthology.orchestra.run.smithi064.stdout:1258 write 0x143760c thru 0x1444440 (0xce35 bytes) 2024-04-24T00:34:37.762 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:37.762 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:37.762 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:37.762 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:37.762 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:37.768 INFO:teuthology.orchestra.run.smithi064.stdout:1259 read 0x8cac205 thru 0x8cb0eb9 (0x4cb5 bytes) 2024-04-24T00:34:37.769 INFO:teuthology.orchestra.run.smithi064.stdout:1261 read 0x3a070ce thru 0x3a0e684 (0x75b7 bytes) 2024-04-24T00:34:37.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:37.789 INFO:teuthology.orchestra.run.smithi064.stdout:1262 write 0xc1c58e7 thru 0xc1d2455 (0xcb6f bytes) 2024-04-24T00:34:37.794 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0xedd99ac 2024-04-24T00:34:37.799 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:37.794+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:37.802 INFO:teuthology.orchestra.run.smithi064.stdout:1264 trunc from 0xc1d2456 to 0xedd99ac 2024-04-24T00:34:37.813 INFO:teuthology.orchestra.run.smithi064.stdout:1265 writesame 0xb825e85 thru 0xb8267ea data_size 0x3(0x966 bytes) 2024-04-24T00:34:37.828 INFO:teuthology.orchestra.run.smithi064.stdout:1266 writesame 0xee4f3ef thru 0xee5cca1 data_size 0x23(0xd8b3 bytes) 2024-04-24T00:34:37.828 INFO:teuthology.orchestra.run.smithi064.stdout:1267 read 0xc86a67f thru 0xc86cf01 (0x2883 bytes) 2024-04-24T00:34:37.828 INFO:teuthology.orchestra.run.smithi064.stdout:1268 punch from 0xb1c1ea0 to 0xb1c2bdb, (0xd3b bytes) 2024-04-24T00:34:37.828 INFO:teuthology.orchestra.run.smithi064.stdout:1269 write 0x3318cd2 thru 0x3323747 (0xaa76 bytes) 2024-04-24T00:34:37.838 INFO:teuthology.orchestra.run.smithi064.stdout:1270 read 0x1bad80b thru 0x1bb7138 (0x992e bytes) 2024-04-24T00:34:37.839 INFO:teuthology.orchestra.run.smithi064.stdout:1271 write 0x48250d2 thru 0x482c4e7 (0x7416 bytes) 2024-04-24T00:34:37.845 INFO:teuthology.orchestra.run.smithi064.stdout:1272 punch from 0xea2772c to 0xea2bea0, (0x4774 bytes) 2024-04-24T00:34:37.845 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:37.857 INFO:teuthology.orchestra.run.smithi064.stdout:1274 trunc from 0xee5cca2 to 0xa7e64da 2024-04-24T00:34:37.878 INFO:teuthology.orchestra.run.smithi064.stdout:1275 writesame 0x5ed2b3 thru 0x5f40a5 data_size 0x1(0x6df3 bytes) 2024-04-24T00:34:37.899 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:37.894+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.977874811, 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.910 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:37.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:37.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:37.916 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.523646077000002, 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024520601, 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 117 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380634+0000", 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380635+0000", 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380642+0000", 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7700000000000004e-06 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380654+0000", 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1686e-05 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.380666+0000", 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2150000000000001e-05 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399787+0000", 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019121461999999999 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.399921+0000", 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134043 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.400127+0000", 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020607500000000001 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.401186+0000", 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001058738 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.404952+0000", 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037661779999999998 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405042+0000", 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9889999999999995e-05 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.405155+0000", 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112812 2024-04-24T00:34:37.921 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4984 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.310530864, 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024665092, 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4984 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4e-07 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593755+0000", 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6829999999999999e-06 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593765+0000", 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7140000000000005e-06 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593773+0000", 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7829999999999999e-06 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609097+0000", 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015324669000000001 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609254+0000", 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015632499999999999 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609918+0000", 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066479400000000002 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.615678+0000", 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057599119999999998 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618123+0000", 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024450909999999999 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618280+0000", 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156706 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618415+0000", 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134475 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.927 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5029 3.f 3:f41ea3ee:::rbd_data.115ed6c33904.0000000000000006:head [zero 262144~262144,zero 786432~65536,zero 917504~39917,write 957421~8409 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.295079499, 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024904274000000001, 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5029 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609208+0000", 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8959999999999997e-06 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609285+0000", 2024-04-24T00:34:37.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6738999999999998e-05 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609308+0000", 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3896999999999998e-05 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609323+0000", 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4543e-05 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622194+0000", 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012870985 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622559+0000", 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036503000000000002 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.623244+0000", 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068464100000000005 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.629548+0000", 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063044329999999999 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.633948+0000", 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044003250000000001 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634037+0000", 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9109000000000004e-05 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634105+0000", 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7675999999999994e-05 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5037 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.610513+0000", 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.293767210999999, 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024006709000000001, 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5037 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610513+0000", 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610513+0000", 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610516+0000", 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7099999999999999e-06 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610530+0000", 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3956e-05 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610560+0000", 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0236999999999999e-05 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.610581+0000", 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0622999999999999e-05 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.624806+0000", 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014225193000000001 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.625000+0000", 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019424500000000001 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.625799+0000", 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00079852000000000005 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.633823+0000", 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080246020000000005 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634233+0000", 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000410169 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634405+0000", 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017113499999999999 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634520+0000", 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115319 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5305 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.685285+0000", 2024-04-24T00:34:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.218995719999999, 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023678181999999999, 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5305 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685285+0000", 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685285+0000", 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685286+0000", 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-06 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685293+0000", 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7270000000000001e-06 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685308+0000", 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5216e-05 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685317+0000", 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6299999999999993e-06 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.693625+0000", 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083071580000000003 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694495+0000", 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00087003500000000001 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695368+0000", 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00087324600000000005 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.699580+0000", 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042119280000000002 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.708785+0000", 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092054130000000008 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.708871+0000", 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5377000000000006e-05 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.708963+0000", 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2281999999999995e-05 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5306 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.218943899999999, 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025510631999999998, 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5306 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685337+0000", 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685342+0000", 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4560000000000002e-06 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685352+0000", 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840000000000006e-06 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685359+0000", 2024-04-24T00:34:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7100000000000007e-06 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696262+0000", 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010902604 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696476+0000", 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021439300000000001 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.697460+0000", 2024-04-24T00:34:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00098361900000000003 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.706972+0000", 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095118609999999999 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710681+0000", 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037093619999999999 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710762+0000", 2024-04-24T00:34:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0374e-05 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710847+0000", 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5478999999999999e-05 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10360 3.11 3:8fda792a:::rbd_data.115e380d621c.0000000000000177:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.452889932000001, 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042632046999999999, 2024-04-24T00:34:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10360 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451393+0000", 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5600000000000001e-06 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451406+0000", 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2984e-05 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451445+0000", 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9045999999999999e-05 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451461+0000", 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5786000000000001e-05 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451608+0000", 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014694900000000001 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451914+0000", 2024-04-24T00:34:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306657 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002601744 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493753+0000", 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039236409 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493841+0000", 2024-04-24T00:34:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8448000000000004e-05 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493931+0000", 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0198999999999995e-05 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494023+0000", 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1265000000000001e-05 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10360 3.11 e40/21 ondisk, result = 0)", 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.442024639, 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031780810999999999, 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462258+0000", 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2539999999999999e-06 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462266+0000", 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8029999999999997e-06 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462294+0000", 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8637e-05 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462314+0000", 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493814+0000", 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031500133 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493831+0000", 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7357e-05 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494037+0000", 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020520400000000001 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.045909123000001, 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.044660374999999, 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10590 3.13 e41/21 ondisk, result = 0)", 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.033414790999998, 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042592342999999998, 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870868+0000", 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1730000000000002e-06 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870876+0000", 2024-04-24T00:34:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2069999999999995e-06 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870900+0000", 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.404e-05 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870914+0000", 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.366e-05 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913167+0000", 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042252855999999998 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913184+0000", 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7779e-05 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913458+0000", 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027362799999999999 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10592 3.16 e41/21 ondisk, result = 0)", 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.028998195, 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044174204000000002, 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875284+0000", 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1050000000000002e-06 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875292+0000", 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3200000000000002e-06 2024-04-24T00:34:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875319+0000", 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7382999999999999e-05 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875334+0000", 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4786e-05 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919179+0000", 2024-04-24T00:34:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043845151999999998 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919195+0000", 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5846999999999999e-05 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919456+0000", 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026161100000000003 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.887036270999999, 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:34:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:34:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:34:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:34:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.873541727999999, 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:34:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21553 3.7 3:e2dbd82a:::rbd_data.115eeb9272a5.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.808058623000001, 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029891540000000001, 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21553 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096223+0000", 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3570000000000001e-06 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096230+0000", 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4080000000000003e-06 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096244+0000", 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4236999999999999e-05 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096251+0000", 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6569999999999997e-06 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096335+0000", 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3997000000000002e-05 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096473+0000", 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013718199999999999 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.097939+0000", 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014665570000000001 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125952+0000", 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028013198 2024-04-24T00:34:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125977+0000", 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4613999999999998e-05 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126035+0000", 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7966e-05 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126113+0000", 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8367000000000004e-05 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.611760613, 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:37.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:37.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21613 3.7 e52/44 ondisk, result = 0)", 2024-04-24T00:34:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.603926674, 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044307553999999999, 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300355+0000", 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4610000000000001e-06 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300362+0000", 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4930000000000003e-06 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300382+0000", 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9887000000000001e-05 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300394+0000", 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2799000000000001e-05 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344558+0000", 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044164017999999999 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344589+0000", 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.027e-05 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344661+0000", 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2626000000000006e-05 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.35817447, 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:34:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:34:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.330500606, 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:34:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:34:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:34:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:38.013 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:38.015 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:38.012+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:38.015 INFO:teuthology.orchestra.run.smithi064.stdout:1276 punch from 0x9ce4fb3 to 0x9cee12a, (0x9177 bytes) 2024-04-24T00:34:38.015 INFO:teuthology.orchestra.run.smithi064.stdout:1277 write 0x32bd816 thru 0x32c3cf7 (0x64e2 bytes) 2024-04-24T00:34:38.015 INFO:teuthology.orchestra.run.smithi064.stdout:1278 punch from 0x1be41a3 to 0x1be7d60, (0x3bbd bytes) 2024-04-24T00:34:38.015 INFO:teuthology.orchestra.run.smithi064.stdout:1279 read 0x67e595 thru 0x68c6f9 (0xe165 bytes) 2024-04-24T00:34:38.015 INFO:teuthology.orchestra.run.smithi064.stdout:1281 read 0x86fba2e thru 0x870625f (0xa832 bytes) 2024-04-24T00:34:38.015 INFO:teuthology.orchestra.run.smithi064.stdout:1282 punch from 0x16b577e to 0x16bd25e, (0x7ae0 bytes) 2024-04-24T00:34:38.015 INFO:teuthology.orchestra.run.smithi064.stdout:1284 trunc from 0xa7e64da to 0x47e15e2 2024-04-24T00:34:38.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:38.022 INFO:teuthology.orchestra.run.smithi064.stdout:1286 write 0xd814a72 thru 0xd81e4b5 (0x9a44 bytes) 2024-04-24T00:34:38.026 INFO:teuthology.orchestra.run.smithi064.stdout:1287 read 0x71ccff8 thru 0x71cd930 (0x939 bytes) 2024-04-24T00:34:38.026 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:38.031 INFO:teuthology.orchestra.run.smithi064.stdout:1289 trunc from 0xd81e4b6 to 0xb9c33e0 2024-04-24T00:34:38.047 INFO:teuthology.orchestra.run.smithi064.stdout:1291 read 0x97b5c75 thru 0x97c3852 (0xdbde bytes) 2024-04-24T00:34:38.048 INFO:teuthology.orchestra.run.smithi064.stdout:1292 punch from 0x942d78a to 0x9435c8f, (0x8505 bytes) 2024-04-24T00:34:38.048 INFO:teuthology.orchestra.run.smithi064.stdout:1293 punch from 0x73affe8 to 0x73be07a, (0xe092 bytes) 2024-04-24T00:34:38.048 INFO:teuthology.orchestra.run.smithi064.stdout:1294 read 0x35ae9f1 thru 0x35b6921 (0x7f31 bytes) 2024-04-24T00:34:38.051 INFO:teuthology.orchestra.run.smithi064.stdout:1297 write 0x114089c thru 0x1144d4b (0x44b0 bytes) 2024-04-24T00:34:38.074 INFO:teuthology.orchestra.run.smithi064.stdout:1298 write 0xe7f0405 thru 0xe7f647e (0x607a bytes) 2024-04-24T00:34:38.079 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:38.079 INFO:teuthology.orchestra.run.smithi064.stdout:1300 clone 9 order 21 su 16384 sc 13 2024-04-24T00:34:38.115 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:38.112+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:38.149 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:38.150 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:38.150 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:38.150 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:38.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:38.216 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:38.212+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:38.317 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:38.312+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:38.335 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:38.335 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:38.335 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:38.335 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:38.335 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:38.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:38.418 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:38.414+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:38.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:38.482 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2019 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.948486578999997, 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014241181, 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2019 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530779+0000", 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530780+0000", 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.686e-06 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.507e-06 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530790+0000", 2024-04-24T00:34:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5240000000000002e-06 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544487+0000", 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013696612 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544570+0000", 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2704e-05 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545020+0000", 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044978100000000001 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2020 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.948467993999998, 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014215145, 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2020 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9499999999999998e-07 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530799+0000", 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.013e-06 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4289999999999998e-06 2024-04-24T00:34:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530810+0000", 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4719999999999997e-06 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544812+0000", 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01400174 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544855+0000", 2024-04-24T00:34:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3217999999999999e-05 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545012+0000", 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000156878 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10361 3.1f 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.026490944999999, 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032855711000000003, 2024-04-24T00:34:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10361 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452776+0000", 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.34e-06 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452789+0000", 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2821e-05 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452813+0000", 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3903000000000001e-05 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452825+0000", 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1904e-05 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452918+0000", 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2742000000000001e-05 2024-04-24T00:34:38.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453066+0000", 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014841100000000001 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00144988 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484060+0000", 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029543894000000001 2024-04-24T00:34:38.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485505+0000", 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014454610000000001 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485582+0000", 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6792000000000006e-05 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485630+0000", 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7562999999999998e-05 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10362 3.17 3:ea0e3a17:::rbd_data.115e380d621c.0000000000000179:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.025362113, 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031846629000000001, 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.497 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10362 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453904+0000", 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.23e-07 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453909+0000", 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7989999999999999e-06 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453922+0000", 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2911999999999999e-05 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453930+0000", 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1430000000000001e-06 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454001+0000", 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1239e-05 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454185+0000", 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018420099999999999 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456031+0000", 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018460180000000001 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485548+0000", 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029516936000000001 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485636+0000", 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8568999999999999e-05 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485693+0000", 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6307000000000001e-05 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485749+0000", 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6681999999999997e-05 2024-04-24T00:34:38.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10361 3.1f e40/21 ondisk, result = 0)", 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.020923584999998, 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025729842999999999, 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458343+0000", 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.189e-06 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458348+0000", 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5130000000000004e-06 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458364+0000", 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6235e-05 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458374+0000", 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4220000000000006e-06 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484036+0000", 2024-04-24T00:34:38.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025662124000000001 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484049+0000", 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3382e-05 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484071+0000", 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1977999999999999e-05 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10362 3.17 e40/21 ondisk, result = 0)", 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.017846001999999, 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024141582000000002, 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461421+0000", 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8339999999999999e-06 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461428+0000", 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4939999999999997e-06 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461455+0000", 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6571999999999999e-05 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461468+0000", 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3243e-05 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485528+0000", 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024059515 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485539+0000", 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0933e-05 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485561+0000", 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1991e-05 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.510 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:20360 3.1b 3:de2cd4b6:::rbd_data.115ee4742374.0000000000000022:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.982412248999999, 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100050336, 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 20360 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496855+0000", 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6060000000000001e-06 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496866+0000", 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0308e-05 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496891+0000", 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5717000000000001e-05 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496904+0000", 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2852e-05 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.497012+0000", 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107873 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.497126+0000", 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114431 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.503011+0000", 2024-04-24T00:34:38.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058843769999999997 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596769+0000", 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093757937 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596790+0000", 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1590000000000002e-05 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596851+0000", 2024-04-24T00:34:38.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1042000000000003e-05 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596903+0000", 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1603e-05 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:20360 3.1b e48/21 ondisk, result = 0)", 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.965298173000001, 2024-04-24T00:34:38.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082812384000000003, 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513968+0000", 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3689999999999999e-06 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513973+0000", 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.673e-06 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513990+0000", 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6806000000000001e-05 2024-04-24T00:34:38.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.514003+0000", 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2924e-05 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596727+0000", 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082724848000000004 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596738+0000", 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0501e-05 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596779+0000", 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1263000000000001e-05 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:20564 3.17 3:ec244480:::rbd_data.115ee4742374.000000000000000d:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.461033709000001, 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081361157000000003, 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.520 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 20564 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:38.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018234+0000", 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.43e-06 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018249+0000", 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4892000000000001e-05 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018285+0000", 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6164999999999997e-05 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018301+0000", 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5860000000000001e-05 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018472+0000", 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171602 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018753+0000", 2024-04-24T00:34:38.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000280611 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.022778+0000", 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004025303 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099276+0000", 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076498022999999998 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099401+0000", 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012506299999999999 2024-04-24T00:34:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099503+0000", 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101409 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099592+0000", 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9798999999999999e-05 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:20564 3.17 e49/21 ondisk, result = 0)", 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.444370225, 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064713039, 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034896+0000", 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.53e-06 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034904+0000", 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2230000000000007e-06 2024-04-24T00:34:38.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034935+0000", 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0278e-05 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034949+0000", 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.403e-05 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099371+0000", 2024-04-24T00:34:38.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064422222000000001 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099390+0000", 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8943000000000001e-05 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099608+0000", 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217813 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.529 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21501 3.b 3:d2c9feba:::rbd_data.115eeb9272a5.000000000000001b:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.750651457, 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.140289634, 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21501 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728616+0000", 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3020000000000002e-06 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728626+0000", 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0045e-05 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728648+0000", 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1933000000000002e-05 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728657+0000", 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6129999999999993e-06 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728726+0000", 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8979000000000005e-05 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728846+0000", 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012003 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.733780+0000", 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004933242 2024-04-24T00:34:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868807+0000", 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13502688400000001 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868823+0000", 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6237999999999999e-05 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868863+0000", 2024-04-24T00:34:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0074999999999998e-05 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868903+0000", 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0293000000000002e-05 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21501 3.b e52/44 ondisk, result = 0)", 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.730039286, 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11958963, 2024-04-24T00:34:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749227+0000", 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.663e-06 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749235+0000", 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7470000000000005e-06 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749256+0000", 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1243999999999999e-05 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749268+0000", 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1968e-05 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868685+0000", 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119416835 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868694+0000", 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1139999999999995e-06 2024-04-24T00:34:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868815+0000", 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012105899999999999 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21529 3.1b 3:ddb27a62:::rbd_data.115eeb9272a5.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.538312696, 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04919722, 2024-04-24T00:34:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21529 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940955+0000", 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7310000000000002e-06 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940971+0000", 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6229999999999999e-05 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940999+0000", 2024-04-24T00:34:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7404000000000001e-05 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941012+0000", 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3315e-05 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941093+0000", 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0828999999999995e-05 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941226+0000", 2024-04-24T00:34:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133189 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.945656+0000", 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044297529999999998 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989931+0000", 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044275675 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990003+0000", 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2053999999999998e-05 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990084+0000", 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0507999999999996e-05 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990149+0000", 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5531999999999995e-05 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21529 3.1b e52/21 ondisk, result = 0)", 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.519313050999999, 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030208611, 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959953+0000", 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2720000000000001e-06 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959960+0000", 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4690000000000002e-06 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959979+0000", 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9188999999999999e-05 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959992+0000", 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2833000000000001e-05 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989981+0000", 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029988982000000001 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989994+0000", 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3603e-05 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990160+0000", 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016626299999999999 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21548 3.6 3:631b2dba:::rbd_data.115eeb9272a5.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.436855276999999, 2024-04-24T00:34:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035743242000000001, 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21548 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042411+0000", 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5700000000000001e-07 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042415+0000", 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5839999999999996e-06 2024-04-24T00:34:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042427+0000", 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1504e-05 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042432+0000", 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4319999999999998e-06 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042481+0000", 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8557999999999998e-05 2024-04-24T00:34:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042573+0000", 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2349999999999995e-05 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.045702+0000", 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003128858 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.077952+0000", 2024-04-24T00:34:38.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032250612999999997 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078015+0000", 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2855999999999996e-05 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078092+0000", 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6922000000000007e-05 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078153+0000", 2024-04-24T00:34:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0708000000000002e-05 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21548 3.6 e52/21 ondisk, result = 0)", 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.423107998000001, 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022006369000000001, 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056158+0000", 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056163+0000", 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8300000000000003e-06 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056180+0000", 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6575999999999999e-05 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056188+0000", 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6219999999999998e-06 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.077995+0000", 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021806968999999999 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078007+0000", 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1771e-05 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078163+0000", 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015619900000000001 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22146 3.18 3:1e672920:::rbd_data.115e6a380959.0000000000000004:head [zero 0~262144,write 262144~262144 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.672408903, 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026558528000000001, 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22146 2024-04-24T00:34:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806857+0000", 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.384e-06 2024-04-24T00:34:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807101+0000", 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024388599999999999 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807118+0000", 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6364000000000001e-05 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807127+0000", 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9469999999999992e-06 2024-04-24T00:34:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807208+0000", 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0982999999999994e-05 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807431+0000", 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022303899999999999 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.808060+0000", 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062940900000000004 2024-04-24T00:34:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832854+0000", 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024794415 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833269+0000", 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041483500000000001 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833342+0000", 2024-04-24T00:34:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2755000000000004e-05 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833415+0000", 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2510999999999997e-05 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22147 3.18 3:1b6fdb14:::rbd_data.115e6a380959.0000000000000002:head [write 0~482922 [fadvise_sequential+fadvise_nocache],zero 482922~97123,write 580045~47154 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.672109989, 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028316734999999999, 2024-04-24T00:34:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22147 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807156+0000", 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6499999999999998e-07 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807545+0000", 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038943700000000002 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807561+0000", 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5724999999999999e-05 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807570+0000", 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1679999999999993e-06 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.820773+0000", 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013202667 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.820953+0000", 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018013799999999999 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.821279+0000", 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000326244 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833920+0000", 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012641316999999999 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835320+0000", 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013997219999999999 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835412+0000", 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2283999999999999e-05 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835472+0000", 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9268000000000002e-05 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22148 3.b 3:d51ed8a7:::rbd_data.115e6a380959.0000000000000005:head [write 0~524288 [fadvise_sequential+fadvise_nocache],zero 1048576~524288,zero 5242880~89944,write 5332824~63306 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.6695143659999996, 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01907449, 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22148 2024-04-24T00:34:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809752+0000", 2024-04-24T00:34:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.764e-06 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810284+0000", 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053170400000000003 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810302+0000", 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7723e-05 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810313+0000", 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1486e-05 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810413+0000", 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9906000000000002e-05 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810646+0000", 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023295499999999999 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810942+0000", 2024-04-24T00:34:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000296218 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.823674+0000", 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012731732000000001 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828700+0000", 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050263729999999998 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828771+0000", 2024-04-24T00:34:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1027999999999998e-05 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828825+0000", 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600999999999997e-05 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:22146 3.18 e58/44 ondisk, result = 0)", 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.6657203919999999, 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01931886, 2024-04-24T00:34:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813546+0000", 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813553+0000", 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1369999999999996e-06 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813574+0000", 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1185000000000001e-05 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813584+0000", 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0297e-05 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832839+0000", 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019254335000000001 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832848+0000", 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9609999999999994e-06 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832863+0000", 2024-04-24T00:34:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5722e-05 2024-04-24T00:34:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:38.578 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:38.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:38.580 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:38.580 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:38.580 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:38.574+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:38.592 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:38.592 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:38.614 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:38.614 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:38.633 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:38.633 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:38.669 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:38.669 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:38.680 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:38.676+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:38.704 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:38.704 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:38.725 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:38.725 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:38.730 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:38.730 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:38.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:38.747 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:38.747 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:38.809 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:38.809 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:38.809 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-24T00:34:38.809 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:34:38.809 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:34:38.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-24T00:34:38.810 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:38.806+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:38.865 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:38.865 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:38.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:38.882 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:38.876+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:38.932 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:38.932 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T00:34:38.932 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:38.982 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:38.980+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:38.999 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:38.999 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:39.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:39.083 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:39.078+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:39.138 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:39.138 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:39.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:39.184 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:39.180+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:39.284 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:39.278+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:39.323 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:39.323 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:39.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:39.385 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:39.380+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:39.486 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:39.482+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:39.506 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:39.506 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:39.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:39.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:34:39.590 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:39.584+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:39.689 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:39.682+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:39.715 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:39.715 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:39.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:39.789 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:39.784+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:39.889 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:39.886+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:39.965 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:39.965 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:39.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:40.029 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:40.026+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:40.092 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:40.088+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:40.180 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:40.180 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:40.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:40.193 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:40.188+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:40.294 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:40.288+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:40.314 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:40.314 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:40.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:40.415 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:40.412+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:40.495 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:40.495 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:40.496 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:40.492+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:40.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:40.585 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:34:40.597 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:40.592+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:40.678 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:40.678 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:40.692 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:40.735 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:40.732+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:40.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:40.775 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-04-24T00:34:40.798 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:40.794+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:40.861 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:40.861 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:40.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:40.899 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:40.894+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:40.911 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:40.911 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:40.928 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:40.929 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:40.945 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:40.946 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:40.962 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:40.962 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:40.982 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:40.982 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:40.991 INFO:teuthology.orchestra.run.smithi064.stdout:1302 read 0x87e5f35 thru 0x87eb4e0 (0x55ac bytes) 2024-04-24T00:34:40.992 INFO:teuthology.orchestra.run.smithi064.stdout:1303 punch from 0x83d6304 to 0x83d6a38, (0x734 bytes) 2024-04-24T00:34:40.997 INFO:teuthology.orchestra.run.smithi064.stdout:1304 write 0xb634dc7 thru 0xb643068 (0xe2a2 bytes) 2024-04-24T00:34:40.998 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:40.998 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:40.999 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:40.994+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:41.006 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:41.006 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:41.018 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:41.021 INFO:teuthology.orchestra.run.smithi064.stdout:1305 trunc from 0xe7f647f to 0xb52cd9a 2024-04-24T00:34:41.029 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:41.030 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:41.052 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:41.053 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:41.053 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.053 INFO:teuthology.orchestra.run.smithi064.stdout:1309 read 0x47108f3 thru 0x471f95b (0xf069 bytes) 2024-04-24T00:34:41.055 INFO:teuthology.orchestra.run.smithi064.stdout:1310 punch from 0x9ec7751 to 0x9ecc8d0, (0x517f bytes) 2024-04-24T00:34:41.055 INFO:teuthology.orchestra.run.smithi064.stdout:1311 punch from 0x1a28607 to 0x1a355f5, (0xcfee bytes) 2024-04-24T00:34:41.055 INFO:teuthology.orchestra.run.smithi064.stdout:1313 read 0x12e96cb thru 0x12f4ec5 (0xb7fb bytes) 2024-04-24T00:34:41.056 INFO:teuthology.orchestra.run.smithi064.stdout:1314 write 0x9a3f6ca thru 0x9a3f760 (0x97 bytes) 2024-04-24T00:34:41.058 INFO:teuthology.orchestra.run.smithi064.stdout:1317 punch from 0x112824e to 0x1130abc, (0x886e bytes) 2024-04-24T00:34:41.058 INFO:teuthology.orchestra.run.smithi064.stdout:1319 read 0x9aebd2e thru 0x9af885b (0xcb2e bytes) 2024-04-24T00:34:41.058 INFO:teuthology.orchestra.run.smithi064.stdout:1320 write 0x9ee9a08 thru 0x9ef6bf3 (0xd1ec bytes) 2024-04-24T00:34:41.062 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.063 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.063 INFO:teuthology.orchestra.run.smithi064.stdout:1323 read 0x137cdce thru 0x138c792 (0xf9c5 bytes) 2024-04-24T00:34:41.064 INFO:teuthology.orchestra.run.smithi064.stdout:1324 read 0x3ac35a1 thru 0x3ad275d (0xf1bd bytes) 2024-04-24T00:34:41.065 INFO:teuthology.orchestra.run.smithi064.stdout:1328 read 0x6e5f0a6 thru 0x6e6d11c (0xe077 bytes) 2024-04-24T00:34:41.066 INFO:teuthology.orchestra.run.smithi064.stdout:1329 write 0x8989d47 thru 0x8990ce3 (0x6f9d bytes) 2024-04-24T00:34:41.070 INFO:teuthology.orchestra.run.smithi064.stdout:1330 writesame 0x7f66877 thru 0x7f7021f data_size 0x1(0x99a9 bytes) 2024-04-24T00:34:41.117 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:41.114+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:41.142 INFO:teuthology.orchestra.run.smithi064.stdout:1331 punch from 0x158e700 to 0x15953e1, (0x6ce1 bytes) 2024-04-24T00:34:41.142 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.142 INFO:teuthology.orchestra.run.smithi064.stdout:1333 read 0x7f12cb7 thru 0x7f22442 (0xf78c bytes) 2024-04-24T00:34:41.147 INFO:teuthology.orchestra.run.smithi064.stdout:1334 trunc from 0xb52cd9a to 0xd39ebe4 2024-04-24T00:34:41.156 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:41.156 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:41.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:41.156 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:41.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.156 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:34:41.156 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.442868235000001, 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:41.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:34:41.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:34:41.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.44277031, 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:41.162 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:34:41.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:34:41.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:34:41.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21662 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:41.168 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.363019908, 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082640219999999993, 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21662 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788216+0000", 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788217+0000", 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788221+0000", 2024-04-24T00:34:41.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0470000000000004e-06 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788234+0000", 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3138e-05 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.788243+0000", 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1759999999999993e-06 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791802+0000", 2024-04-24T00:34:41.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003559347 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.791936+0000", 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134349 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.796480+0000", 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004544022 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22095 3.1c 3:3fa7153c:::rbd_data.115eeb9272a5.0000000000000002:11 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.794499+0000", 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.356737055, 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070792279999999999, 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22095 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794499+0000", 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794499+0000", 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794500+0000", 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2799999999999995e-07 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794503+0000", 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5750000000000001e-06 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794515+0000", 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.794520+0000", 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6300000000000003e-06 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.800825+0000", 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063040769999999999 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.801118+0000", 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029312099999999997 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.801578+0000", 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00046063300000000001 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22145 3.1d 3:bf5208ff:::rbd_data.115e6a380959.0000000000000001:head [write 0~524288 [fadvise_sequential+fadvise_nocache],zero 1048576~311524,write 1360100~9561 [fadvise_sequential+fadvise_nocache],zero 2621440~524288,zero 6815744~1717,write 6817461~5360 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.804836+0000", 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3463997259999996, 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015668827, 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22145 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.804836+0000", 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.804836+0000", 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.804837+0000", 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6899999999999996e-07 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805641+0000", 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080407199999999997 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805653+0000", 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1240000000000001e-05 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805660+0000", 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5020000000000003e-06 2024-04-24T00:34:41.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805736+0000", 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5724999999999997e-05 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.805889+0000", 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015369299999999999 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.806194+0000", 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030478699999999999 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820040+0000", 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013846089000000001 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820279+0000", 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023887499999999999 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820435+0000", 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015621100000000001 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820505+0000", 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9764000000000005e-05 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22149 3.c 3:3770836c:::rbd_data.115e6a380959.0000000000000006:head [write 0~524288 [fadvise_sequential+fadvise_nocache],zero 524288~524288,zero 1572864~524288,zero 5767168~1048576,zero 8388608~422408,write 8811016~8118 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.810003+0000", 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3412328920000007, 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01478607, 2024-04-24T00:34:41.182 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22149 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810003+0000", 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810003+0000", 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810005+0000", 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2050000000000001e-06 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810532+0000", 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00052670099999999997 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810553+0000", 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1285999999999999e-05 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810566+0000", 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2551000000000001e-05 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810682+0000", 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116123 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.810975+0000", 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029288900000000001 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811371+0000", 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039598999999999998 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824401+0000", 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013030168 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824615+0000", 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021383400000000001 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824706+0000", 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1043000000000002e-05 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824789+0000", 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3280000000000002e-05 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.187 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22150 3.c 3:36ffc5a2:::rbd_data.115e6a380959.0000000000000003:head [write 0~262144 [fadvise_sequential+fadvise_nocache],zero 262144~196158,write 458302~37038 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.811490+0000", 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3397464689999996, 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016278401000000001, 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22150 2024-04-24T00:34:41.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811490+0000", 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811490+0000", 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811491+0000", 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1119999999999999e-06 2024-04-24T00:34:41.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811826+0000", 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000335547 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811837+0000", 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0727000000000001e-05 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.811848+0000", 2024-04-24T00:34:41.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1355000000000001e-05 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820484+0000", 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086351369999999993 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820601+0000", 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011789400000000001 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.820805+0000", 2024-04-24T00:34:41.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020369799999999999 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825543+0000", 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047380230000000001 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.827715+0000", 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002172186 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.827746+0000", 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0229000000000001e-05 2024-04-24T00:34:41.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.827768+0000", 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2493e-05 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22152 3.1c 3:3c0f4810:::rbd_data.115e6a380959.0000000000000008:head [zero 0~103219,write 103219~11656 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.813060+0000", 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3381757570000001, 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013815094, 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.193 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22152 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813060+0000", 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813060+0000", 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813061+0000", 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813078+0000", 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7125e-05 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813089+0000", 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0794e-05 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.813114+0000", 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4272000000000001e-05 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824337+0000", 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011223568 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824486+0000", 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148942 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824613+0000", 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012721400000000001 2024-04-24T00:34:41.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826048+0000", 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001434301 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826808+0000", 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00076005199999999999 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826849+0000", 2024-04-24T00:34:41.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1198000000000001e-05 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.826875+0000", 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6625e-05 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22164 3.c 3:30fd3d95:::rbd_data.115eeb9272a5.000000000000000d:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.817271+0000", 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3339650330000001, 2024-04-24T00:34:41.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075233560000000001, 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22164 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817271+0000", 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817271+0000", 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817272+0000", 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6600000000000005e-07 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817276+0000", 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6720000000000002e-06 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817283+0000", 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6879999999999999e-06 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817289+0000", 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6339999999999998e-06 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824455+0000", 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0071660409999999997 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.824794+0000", 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033945500000000001 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22168 3.1c 3:381c9e4b:::rbd_data.115eeb9272a5.0000000000000015:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.817343+0000", 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3338927530000007, 2024-04-24T00:34:41.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082038319999999994, 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22168 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817343+0000", 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817343+0000", 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817344+0000", 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817348+0000", 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8929999999999998e-06 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817355+0000", 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6610000000000003e-06 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817360+0000", 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6080000000000003e-06 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825264+0000", 2024-04-24T00:34:41.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079035140000000004 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825547+0000", 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028333299999999998 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22182 3.c 3:3159bf60:::rbd_data.115eeb9272a5.000000000000002c:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.207 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.817414+0000", 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3338225179999998, 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007932616, 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22182 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817414+0000", 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817414+0000", 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817414+0000", 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5499999999999998e-07 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817416+0000", 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817422+0000", 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0859999999999998e-06 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817428+0000", 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2639999999999997e-06 2024-04-24T00:34:41.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825222+0000", 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0077935230000000001 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825346+0000", 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124265 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22188 3.c 3:3391bdb6:::rbd_data.115eeb9272a5.0000000000000037:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.211 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.817481+0000", 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3337550979999993, 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080402930000000004, 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22188 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817481+0000", 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817481+0000", 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817481+0000", 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5499999999999998e-07 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817484+0000", 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2630000000000002e-06 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817494+0000", 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0689999999999999e-05 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.817500+0000", 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1050000000000003e-06 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825401+0000", 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079010790000000001 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825521+0000", 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120701 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22190 3.1c 3:3fa7153c:::rbd_data.115eeb9272a5.0000000000000002:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:32.818453+0000", 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.3327828890000006, 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072144100000000001, 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.215 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22190 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818453+0000", 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818453+0000", 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818454+0000", 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7499999999999999e-07 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818458+0000", 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3880000000000002e-06 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818472+0000", 2024-04-24T00:34:41.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.398e-05 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.818480+0000", 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2729999999999997e-06 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825548+0000", 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070680250000000003 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825605+0000", 2024-04-24T00:34:41.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6907999999999998e-05 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:32.825668+0000", 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2960999999999996e-05 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22410 3.1f 3:ff4fbbf8:::rbd_data.115e6a380959.000000000000000c:head [write 5497134~16259] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.572467+0000", 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.5787692799999999, 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095743519999999995, 2024-04-24T00:34:41.219 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22410 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.572467+0000", 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.572467+0000", 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.572470+0000", 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0400000000000001e-06 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.572528+0000", 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8072999999999998e-05 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.572564+0000", 2024-04-24T00:34:41.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5902999999999998e-05 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.572580+0000", 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6589999999999999e-05 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.572727+0000", 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000146985 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.572740+0000", 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.29e-05 2024-04-24T00:34:41.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.576089+0000", 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003348317 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.576294+0000", 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000205572 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.576519+0000", 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022430099999999999 2024-04-24T00:34:41.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.579081+0000", 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025628510000000001 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.581854+0000", 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002772596 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.581962+0000", 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108205 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.582041+0000", 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9018999999999997e-05 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22507 3.1f 3:ff4fbbf8:::rbd_data.115e6a380959.000000000000000c:head [write 1275404~52789] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.758955+0000", 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.3922811089999998, 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079088950000000009, 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22507 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.758955+0000", 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.758955+0000", 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.758956+0000", 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0690000000000001e-06 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.759049+0000", 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2577000000000005e-05 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.759064+0000", 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5653000000000001e-05 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.759073+0000", 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6929999999999996e-06 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.759188+0000", 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115401 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.759333+0000", 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144232 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:41.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.759551+0000", 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218645 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.766156+0000", 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066046949999999998 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.766678+0000", 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00052156200000000005 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.766775+0000", 2024-04-24T00:34:41.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7645000000000004e-05 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.766864+0000", 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8722999999999997e-05 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22547 3.c 3:36ffc5a2:::rbd_data.115e6a380959.0000000000000003:head [truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.857130+0000", 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.2941060740000001, 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074745039999999999, 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.230 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22547 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857130+0000", 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857130+0000", 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857131+0000", 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2399999999999998e-07 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857136+0000", 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5949999999999998e-06 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857144+0000", 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6680000000000001e-06 2024-04-24T00:34:41.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857150+0000", 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0850000000000002e-06 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857322+0000", 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000171712 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857400+0000", 2024-04-24T00:34:41.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8617999999999998e-05 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857533+0000", 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000132688 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.862509+0000", 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049761299999999996 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864411+0000", 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00190219 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864524+0000", 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113089 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864605+0000", 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0105000000000006e-05 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22550 3.c 3:3770836c:::rbd_data.115e6a380959.0000000000000006:head [truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.857175+0000", 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.2940611450000001, 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075900639999999997, 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.236 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22550 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857175+0000", 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857175+0000", 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857176+0000", 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7000000000000005e-07 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857179+0000", 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7589999999999999e-06 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857187+0000", 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5449999999999998e-06 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857192+0000", 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1080000000000001e-06 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.858243+0000", 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001051504 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.858339+0000", 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5501999999999995e-05 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.858505+0000", 2024-04-24T00:34:41.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016620299999999999 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.862609+0000", 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041040290000000004 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864620+0000", 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002010979 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864698+0000", 2024-04-24T00:34:41.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8206000000000003e-05 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864765+0000", 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6659e-05 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22552 3.1c 3:3c0f4810:::rbd_data.115e6a380959.0000000000000008:head [truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.857201+0000", 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.2940354799999998, 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0077138750000000002, 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.241 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22552 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857201+0000", 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857201+0000", 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857201+0000", 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857204+0000", 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1820000000000002e-06 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857211+0000", 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3450000000000002e-06 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857215+0000", 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7620000000000003e-06 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.858002+0000", 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00078667600000000004 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.858224+0000", 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000222048 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.858550+0000", 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032561100000000001 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.862338+0000", 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037886059999999999 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864777+0000", 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024383489999999998 2024-04-24T00:34:41.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864849+0000", 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2298999999999993e-05 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864915+0000", 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5510000000000001e-05 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22556 3.1f 3:ff4fbbf8:::rbd_data.115e6a380959.000000000000000c:head [truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.857242+0000", 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.2939945110000002, 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.007317367, 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22556 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857242+0000", 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857242+0000", 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857242+0000", 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8599999999999997e-07 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857246+0000", 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6059999999999999e-06 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857253+0000", 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2509999999999997e-06 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857259+0000", 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.502e-06 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857582+0000", 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032315799999999998 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857670+0000", 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7781999999999994e-05 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.857805+0000", 2024-04-24T00:34:41.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013496600000000001 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.860050+0000", 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022448540000000001 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864418+0000", 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043680569999999998 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864499+0000", 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1305999999999997e-05 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.864559+0000", 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0099000000000002e-05 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30064 3.1c 3:3ea50142:::rbd_data.115e8210e51a.000000000000006c:head [delete] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.022794+0000", 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.128441851, 2024-04-24T00:34:41.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086861850000000008, 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30064 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.022794+0000", 2024-04-24T00:34:41.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.022794+0000", 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.022795+0000", 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7599999999999996e-07 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.022799+0000", 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.817e-06 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.022808+0000", 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.081e-06 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.022819+0000", 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1386000000000001e-05 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.024491+0000", 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016715180000000001 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.024613+0000", 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121932 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.024720+0000", 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107596 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.028521+0000", 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038005420000000001 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.031368+0000", 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028473029999999998 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.031440+0000", 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1919e-05 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.031480+0000", 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0315000000000003e-05 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.258 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:41.260 INFO:teuthology.orchestra.run.smithi064.stdout:1335 writesame 0x2473ff5 thru 0x24814c6 data_size 0x1(0xd4d2 bytes) 2024-04-24T00:34:41.260 INFO:teuthology.orchestra.run.smithi064.stdout:1336 read 0xcd878d4 thru 0xcd90662 (0x8d8f bytes) 2024-04-24T00:34:41.260 INFO:teuthology.orchestra.run.smithi064.stdout:1337 trunc from 0xd39ebe4 to 0x9f6aaf2 2024-04-24T00:34:41.260 INFO:teuthology.orchestra.run.smithi064.stdout:1338 trunc from 0x9f6aaf2 to 0x53ee899 2024-04-24T00:34:41.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:41.261 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:41.256+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:41.305 INFO:teuthology.orchestra.run.smithi064.stdout:1342 write 0x6812672 thru 0x6818f0d (0x689c bytes) 2024-04-24T00:34:41.315 INFO:teuthology.orchestra.run.smithi064.stdout:1343 read 0x4c1722f thru 0x4c23b7b (0xc94d bytes) 2024-04-24T00:34:41.322 INFO:teuthology.orchestra.run.smithi064.stdout:1347 trunc from 0x6818f0e to 0x8394f1f 2024-04-24T00:34:41.335 INFO:teuthology.orchestra.run.smithi064.stdout:1349 read 0x6a095e thru 0x6ad7b7 (0xce5a bytes) 2024-04-24T00:34:41.361 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:41.356+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:41.365 INFO:teuthology.orchestra.run.smithi064.stdout:1350 write 0xe04afe6 thru 0xe056e0c (0xbe27 bytes) 2024-04-24T00:34:41.374 INFO:teuthology.orchestra.run.smithi064.stdout:1351 read 0xdd7d5bd thru 0xdd7e923 (0x1367 bytes) 2024-04-24T00:34:41.375 INFO:teuthology.orchestra.run.smithi064.stdout:1352 read 0x794224c thru 0x79441a3 (0x1f58 bytes) 2024-04-24T00:34:41.375 INFO:teuthology.orchestra.run.smithi064.stdout:1353 write 0xaa625b1 thru 0xaa68154 (0x5ba4 bytes) 2024-04-24T00:34:41.387 INFO:teuthology.orchestra.run.smithi064.stdout:1357 punch from 0x7c6215a to 0x7c7056a, (0xe410 bytes) 2024-04-24T00:34:41.388 INFO:teuthology.orchestra.run.smithi064.stdout:1358 read 0x8320307 thru 0x832f626 (0xf320 bytes) 2024-04-24T00:34:41.388 INFO:teuthology.orchestra.run.smithi064.stdout:1360 write 0x3870057 thru 0x387c0f9 (0xc0a3 bytes) 2024-04-24T00:34:41.397 INFO:teuthology.orchestra.run.smithi064.stdout:1361 trunc from 0xe056e0d to 0xec80dd3 2024-04-24T00:34:41.410 INFO:teuthology.orchestra.run.smithi064.stdout:1363 punch from 0x8bac2ca to 0x8bb4bb1, (0x88e7 bytes) 2024-04-24T00:34:41.410 INFO:teuthology.orchestra.run.smithi064.stdout:1365 punch from 0x42c8975 to 0x42d03b3, (0x7a3e bytes) 2024-04-24T00:34:41.410 INFO:teuthology.orchestra.run.smithi064.stdout:1366 read 0x56cab39 thru 0x56d9009 (0xe4d1 bytes) 2024-04-24T00:34:41.412 INFO:teuthology.orchestra.run.smithi064.stdout:1367 read 0xcd53064 thru 0xcd62f97 (0xff34 bytes) 2024-04-24T00:34:41.412 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.412 INFO:teuthology.orchestra.run.smithi064.stdout:1370 punch from 0x889f7f0 to 0x88a1bc4, (0x23d4 bytes) 2024-04-24T00:34:41.412 INFO:teuthology.orchestra.run.smithi064.stdout:1372 read 0x807ec0c thru 0x808c078 (0xd46d bytes) 2024-04-24T00:34:41.412 INFO:teuthology.orchestra.run.smithi064.stdout:1373 write 0x6429be7 thru 0x642a5ca (0x9e4 bytes) 2024-04-24T00:34:41.414 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.437 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:41.437 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:41.437 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:41.437 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:41.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:1375 trunc from 0xec80dd3 to 0x55b05f 2024-04-24T00:34:41.462 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:41.458+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:41.498 INFO:teuthology.orchestra.run.smithi064.stdout:1377 punch from 0x2a270f to 0x2ae62d, (0xbf1e bytes) 2024-04-24T00:34:41.498 INFO:teuthology.orchestra.run.smithi064.stdout:1378 read 0x3cc661 thru 0x3cca1f (0x3bf bytes) 2024-04-24T00:34:41.499 INFO:teuthology.orchestra.run.smithi064.stdout:1381 read 0x40069e thru 0x40361c (0x2f7f bytes) 2024-04-24T00:34:41.500 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.501 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.521 INFO:teuthology.orchestra.run.smithi064.stdout:1386 write 0x3f20fcc thru 0x3f2170e (0x743 bytes) 2024-04-24T00:34:41.527 INFO:teuthology.orchestra.run.smithi064.stdout:1387 read 0x3abdf13 thru 0x3acb259 (0xd347 bytes) 2024-04-24T00:34:41.527 INFO:teuthology.orchestra.run.smithi064.stdout:1388 punch from 0x1fd8627 to 0x1fdea46, (0x641f bytes) 2024-04-24T00:34:41.527 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.527 INFO:teuthology.orchestra.run.smithi064.stdout:1391 punch from 0xd51a32 to 0xd5613d, (0x470b bytes) 2024-04-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi064.stdout:1392 writesame 0x45c1bca thru 0x45c9842 data_size 0x1(0x7c79 bytes) 2024-04-24T00:34:41.542 INFO:teuthology.orchestra.run.smithi064.stdout:1393 punch from 0x2126cd4 to 0x212be8b, (0x51b7 bytes) 2024-04-24T00:34:41.543 INFO:teuthology.orchestra.run.smithi064.stdout:1395 trunc from 0x45c9843 to 0x41ebac0 2024-04-24T00:34:41.563 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:41.560+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:41.598 INFO:teuthology.orchestra.run.smithi064.stdout:1396 write 0xdce6025 thru 0xdceafdb (0x4fb7 bytes) 2024-04-24T00:34:41.606 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.607 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.607 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.607 INFO:teuthology.orchestra.run.smithi064.stdout:1400 punch from 0x71bc2d5 to 0x71bea5e, (0x2789 bytes) 2024-04-24T00:34:41.607 INFO:teuthology.orchestra.run.smithi064.stdout:1401 read 0x684cc7f thru 0x684e3df (0x1761 bytes) 2024-04-24T00:34:41.607 INFO:teuthology.orchestra.run.smithi064.stdout:1403 write 0x4393309 thru 0x4395a46 (0x273e bytes) 2024-04-24T00:34:41.614 INFO:teuthology.orchestra.run.smithi064.stdout:1405 read 0xab81de6 thru 0xab90b53 (0xed6e bytes) 2024-04-24T00:34:41.614 INFO:teuthology.orchestra.run.smithi064.stdout:1406 write 0xb3e093 thru 0xb4020b (0x2179 bytes) 2024-04-24T00:34:41.617 INFO:teuthology.orchestra.run.smithi064.stdout:1407 writesame 0xd185fb9 thru 0xd18c872 data_size 0x23(0x68ba bytes) 2024-04-24T00:34:41.618 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:41.618 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:41.618 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:41.618 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:41.618 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:41.623 INFO:teuthology.orchestra.run.smithi064.stdout:1409 punch from 0x5fe807f to 0x5ff51e4, (0xd165 bytes) 2024-04-24T00:34:41.624 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.624 INFO:teuthology.orchestra.run.smithi064.stdout:1413 read 0x74c6f1 thru 0x7504a0 (0x3db0 bytes) 2024-04-24T00:34:41.625 INFO:teuthology.orchestra.run.smithi064.stdout:1415 read 0x6006fe6 thru 0x6015944 (0xe95f bytes) 2024-04-24T00:34:41.625 INFO:teuthology.orchestra.run.smithi064.stdout:1416 write 0xd8bd8aa thru 0xd8c35f3 (0x5d4a bytes) 2024-04-24T00:34:41.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:41.634 INFO:teuthology.orchestra.run.smithi064.stdout:1417 write 0x2312f7b thru 0x231ad42 (0x7dc8 bytes) 2024-04-24T00:34:41.645 INFO:teuthology.orchestra.run.smithi064.stdout:1419 read 0x6562669 thru 0x656e86f (0xc207 bytes) 2024-04-24T00:34:41.659 INFO:teuthology.orchestra.run.smithi064.stdout:1422 writesame 0xe9194c0 thru 0xe91e569 data_size 0xe(0x50aa bytes) 2024-04-24T00:34:41.664 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:41.658+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:41.691 INFO:teuthology.orchestra.run.smithi064.stdout:1424 trunc from 0xe91e56a to 0x38ff109 2024-04-24T00:34:41.721 INFO:teuthology.orchestra.run.smithi064.stdout:1425 trunc from 0x38ff109 to 0x1a5a46f 2024-04-24T00:34:41.765 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:41.760+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:41.788 INFO:teuthology.orchestra.run.smithi064.stdout:1426 write 0xbe99d81 thru 0xbe9a057 (0x2d7 bytes) 2024-04-24T00:34:41.809 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:41.809 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:41.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:41.809 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:41.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.809 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:5336 3.1d e36/21 3:b83bad9e:::rbd_data.115ed6c33904.0000000000000086:head v 36'3, mlcod=34'1)", 2024-04-24T00:34:41.809 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:41.809 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.110276204000002, 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010615080000000001, 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694760+0000", 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8080000000000001e-06 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694891+0000", 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013035600000000001 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694916+0000", 2024-04-24T00:34:41.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5392e-05 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694930+0000", 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4059999999999999e-05 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695055+0000", 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124499 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695112+0000", 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7853000000000003e-05 2024-04-24T00:34:41.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705273+0000", 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010160626000000001 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705373+0000", 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100486 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10257 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.602632244999999, 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011690176, 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10257 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202404+0000", 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.897e-06 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202412+0000", 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202430+0000", 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8043e-05 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202442+0000", 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439999999999999e-05 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207283+0000", 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048410600000000003 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207552+0000", 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026864100000000002 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.208755+0000", 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012028189999999999 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.211687+0000", 2024-04-24T00:34:41.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029325240000000002 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.213944+0000", 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022567300000000002 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214023+0000", 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8932999999999998e-05 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214093+0000", 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9538999999999999e-05 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10263 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.512196958000001, 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091981449999999996, 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10263 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292839+0000", 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-06 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292846+0000", 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.81e-06 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292862+0000", 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6456e-05 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292870+0000", 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3699999999999995e-06 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296812+0000", 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039413900000000003 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296881+0000", 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9426999999999998e-05 2024-04-24T00:34:41.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.297166+0000", 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028513300000000002 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301682+0000", 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045159029999999999 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301927+0000", 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024475499999999998 2024-04-24T00:34:41.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301980+0000", 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2796000000000001e-05 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.302036+0000", 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5992000000000002e-05 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10288 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.472288831, 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.009564744, 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10288 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332747+0000", 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332769+0000", 2024-04-24T00:34:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1903e-05 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332780+0000", 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1243000000000001e-05 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332789+0000", 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0699999999999996e-06 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336540+0000", 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037510339999999999 2024-04-24T00:34:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336760+0000", 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021991099999999999 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.337418+0000", 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000658127 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.339101+0000", 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016826390000000001 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342150+0000", 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030494850000000002 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342236+0000", 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6212999999999996e-05 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342311+0000", 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4228999999999997e-05 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.829 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10359 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.361854427000001, 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091612789999999996, 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.830 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10359 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443180+0000", 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443182+0000", 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443189+0000", 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1470000000000004e-06 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443206+0000", 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7603000000000001e-05 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.443221+0000", 2024-04-24T00:34:41.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4747e-05 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.448444+0000", 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052232700000000003 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.448605+0000", 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016098499999999999 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.449528+0000", 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092301199999999996 2024-04-24T00:34:41.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451989+0000", 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024603250000000002 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452174+0000", 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018491100000000001 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452265+0000", 2024-04-24T00:34:41.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1283e-05 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.452341+0000", 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6656000000000006e-05 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10365 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.310262611999999, 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098439340000000004, 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.835 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10365 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:41.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494773+0000", 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1399999999999995e-07 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494782+0000", 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3649999999999993e-06 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494794+0000", 2024-04-24T00:34:41.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1606e-05 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494804+0000", 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0013999999999999e-05 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498700+0000", 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038961310000000002 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498969+0000", 2024-04-24T00:34:41.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000269432 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.499771+0000", 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080119200000000005 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.503502+0000", 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037318300000000002 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504304+0000", 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080115000000000004 2024-04-24T00:34:41.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504475+0000", 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017146199999999999 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504616+0000", 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014083800000000001 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10409 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:41.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.228680410999999, 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010506112, 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10409 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:41.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576355+0000", 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576359+0000", 2024-04-24T00:34:41.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3229999999999999e-06 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576372+0000", 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576383+0000", 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1192999999999999e-05 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580539+0000", 2024-04-24T00:34:41.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041557230000000001 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580813+0000", 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027430400000000001 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.582037+0000", 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001223879 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.584201+0000", 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002164408 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586567+0000", 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023657090000000001 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586731+0000", 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016439800000000001 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586860+0000", 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128835 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.845 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10518 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.055607980000001, 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091784659999999997, 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.846 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10518 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749427+0000", 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2699999999999998e-07 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749432+0000", 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2780000000000004e-06 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749444+0000", 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1882000000000001e-05 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.749451+0000", 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1890000000000001e-06 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.753811+0000", 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043599609999999999 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.753978+0000", 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016679100000000001 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.754514+0000", 2024-04-24T00:34:41.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00053602499999999998 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758097+0000", 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035836169999999999 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758429+0000", 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033133099999999998 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758527+0000", 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8919000000000006e-05 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.758605+0000", 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7645999999999995e-05 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10645 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.777676328999998, 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010437706999999999, 2024-04-24T00:34:41.851 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10645 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:41.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027359+0000", 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.06e-06 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027365+0000", 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2769999999999996e-06 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027380+0000", 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5537e-05 2024-04-24T00:34:41.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027389+0000", 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5299999999999996e-06 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.030838+0000", 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003449232 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031115+0000", 2024-04-24T00:34:41.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000277254 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031925+0000", 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080932599999999997 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.035618+0000", 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003693647 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037627+0000", 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020091390000000001 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037720+0000", 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2775000000000005e-05 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037796+0000", 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.593e-05 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10719 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.626505162000001, 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010945605000000001, 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10719 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178530+0000", 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1800000000000005e-07 2024-04-24T00:34:41.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178535+0000", 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5129999999999998e-06 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178544+0000", 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2359999999999993e-06 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178550+0000", 2024-04-24T00:34:41.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.421e-06 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181685+0000", 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031346989999999999 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181965+0000", 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027960399999999997 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.182619+0000", 2024-04-24T00:34:41.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065424799999999996 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.186425+0000", 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038062690000000001 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189312+0000", 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028871809999999999 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189407+0000", 2024-04-24T00:34:41.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4345000000000006e-05 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189475+0000", 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8270999999999997e-05 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10739 3.10 3:0a996d09:::rbd_data.115e380d621c.0000000000000199:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:41.862 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.583608659999999, 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092681780000000002, 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10739 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:41.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221428+0000", 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0949999999999998e-06 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221438+0000", 2024-04-24T00:34:41.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9310000000000003e-06 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221463+0000", 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5023000000000001e-05 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221477+0000", 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3591000000000001e-05 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221633+0000", 2024-04-24T00:34:41.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156565 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.222026+0000", 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039234399999999998 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.223749+0000", 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017237109999999999 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230482+0000", 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067323670000000004 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230517+0000", 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981999999999998e-05 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230610+0000", 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230694+0000", 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4393000000000003e-05 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10768 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.522377756000001, 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092323920000000007, 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10768 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:41.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282658+0000", 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1799999999999999e-06 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282664+0000", 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6339999999999998e-06 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282676+0000", 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2320000000000001e-05 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282687+0000", 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1103e-05 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284651+0000", 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00196408 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284827+0000", 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175768 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.285720+0000", 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089352799999999999 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289504+0000", 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003783813 2024-04-24T00:34:41.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291726+0000", 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022220339999999999 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291814+0000", 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7425999999999999e-05 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291889+0000", 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5506000000000005e-05 2024-04-24T00:34:41.873 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10769 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.522293179000002, 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012818598, 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.874 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10769 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282742+0000", 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282747+0000", 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0329999999999999e-06 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282759+0000", 2024-04-24T00:34:41.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1687e-05 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282768+0000", 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.286885+0000", 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041171139999999998 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.287035+0000", 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149968 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.288104+0000", 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010690540000000001 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295331+0000", 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072273800000000003 2024-04-24T00:34:41.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295418+0000", 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295493+0000", 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4825000000000003e-05 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295560+0000", 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7191000000000001e-05 2024-04-24T00:34:41.879 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10770 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.522253930000002, 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013929258999999999, 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.880 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10770 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282782+0000", 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9000000000000005e-07 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282786+0000", 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1330000000000001e-06 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282795+0000", 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-06 2024-04-24T00:34:41.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282803+0000", 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8499999999999994e-06 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289625+0000", 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068213689999999999 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289770+0000", 2024-04-24T00:34:41.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014539899999999999 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.290476+0000", 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070591399999999998 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295352+0000", 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048766089999999996 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296546+0000", 2024-04-24T00:34:41.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011937580000000001 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296638+0000", 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2179e-05 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296710+0000", 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1527999999999997e-05 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.522220829999998, 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.886 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:41.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:34:41.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:34:41.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:34:41.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.522184816999999, 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.894 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:34:41.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.272780423, 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:34:41.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:34:41.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.238743370000002, 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:34:41.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.909 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.64801740600000002, 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:41.910 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:34:41.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.915 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.63876128099999996, 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:41.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:34:41.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:41.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:41.919 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:41.920 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:41.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:1428 writesame 0xe8136ae thru 0xe81c303 data_size 0x47(0x8c56 bytes) 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:1429 write 0xa00e4ae thru 0xa01004c (0x1b9f bytes) 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:1431 read 0x4f8c88c thru 0x4f9134b (0x4ac0 bytes) 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:1434 punch from 0x70741cc to 0x707911b, (0x4f4f bytes) 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:1435 read 0x6d66fb4 thru 0x6d74314 (0xd361 bytes) 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:1436 read 0xa884a8b thru 0xa893aa1 (0xf017 bytes) 2024-04-24T00:34:41.923 INFO:teuthology.orchestra.run.smithi064.stdout:1438 write 0xaa952d thru 0xaae6f0 (0x51c4 bytes) 2024-04-24T00:34:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:1440 read 0xa2b14b5 thru 0xa2bcfe2 (0xbb2e bytes) 2024-04-24T00:34:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:1441 punch from 0x615ba96 to 0x6160557, (0x4ac1 bytes) 2024-04-24T00:34:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:1442 write 0xbaa4276 thru 0xbaaa38e (0x6119 bytes) 2024-04-24T00:34:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:1443 write 0xcf7809c thru 0xcf87f16 (0xfe7b bytes) 2024-04-24T00:34:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:1445 punch from 0x1c8307 to 0x1cee46, (0x6b3f bytes) 2024-04-24T00:34:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:1446 punch from 0xbed5fb8 to 0xbedb7d8, (0x5820 bytes) 2024-04-24T00:34:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:1447 trunc from 0xe81c304 to 0x561104e 2024-04-24T00:34:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:1449 write 0x7d01bea thru 0x7d0de97 (0xc2ae bytes) 2024-04-24T00:34:41.924 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:41.914+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:41.953 INFO:teuthology.orchestra.run.smithi064.stdout:1450 trunc from 0x7d0de98 to 0x1d3d023 2024-04-24T00:34:41.983 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:42.020 INFO:teuthology.orchestra.run.smithi064.stdout:1452 writesame 0xa71c85a thru 0xa725693 data_size 0x37(0x8e3a bytes) 2024-04-24T00:34:42.020 INFO:teuthology.orchestra.run.smithi064.stdout:1453 write 0x3752191 thru 0x3754240 (0x20b0 bytes) 2024-04-24T00:34:42.022 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:42.016+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:42.027 INFO:teuthology.orchestra.run.smithi064.stdout:1454 read 0x3ac83a1 thru 0x3ad6be8 (0xe848 bytes) 2024-04-24T00:34:42.028 INFO:teuthology.orchestra.run.smithi064.stdout:1455 write 0x9aff1c thru 0x9b3e4d (0x3f32 bytes) 2024-04-24T00:34:42.033 INFO:teuthology.orchestra.run.smithi064.stdout:1456 punch from 0x78a2c2c to 0x78a8410, (0x57e4 bytes) 2024-04-24T00:34:42.033 INFO:teuthology.orchestra.run.smithi064.stdout:1458 write 0xa2d4237 thru 0xa2d7433 (0x31fd bytes) 2024-04-24T00:34:42.039 INFO:teuthology.orchestra.run.smithi064.stdout:1460 read 0x933c617 thru 0x93496f4 (0xd0de bytes) 2024-04-24T00:34:42.039 INFO:teuthology.orchestra.run.smithi064.stdout:1461 writesame 0xa3db358 thru 0xa3e4924 data_size 0x3(0x95cd bytes) 2024-04-24T00:34:42.060 INFO:teuthology.orchestra.run.smithi064.stdout:1463 punch from 0xd163a9 to 0xd1df95, (0x7bec bytes) 2024-04-24T00:34:42.060 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:42.060 INFO:teuthology.orchestra.run.smithi064.stdout:1465 writesame 0xa12f1da thru 0xa135865 data_size 0x2(0x668c bytes) 2024-04-24T00:34:42.089 INFO:teuthology.orchestra.run.smithi064.stdout:1466 read 0x6e6994 thru 0x6eda1b (0x7088 bytes) 2024-04-24T00:34:42.100 INFO:teuthology.orchestra.run.smithi064.stdout:1469 trunc from 0xa725694 to 0x7418e42 2024-04-24T00:34:42.101 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:42.101 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:42.101 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:42.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:42.122 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:42.118+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:42.126 INFO:teuthology.orchestra.run.smithi064.stdout:1470 write 0x6d977a7 thru 0x6d9852c (0xd86 bytes) 2024-04-24T00:34:42.135 INFO:teuthology.orchestra.run.smithi064.stdout:1471 punch from 0x604f75e to 0x6057159, (0x79fb bytes) 2024-04-24T00:34:42.168 INFO:teuthology.orchestra.run.smithi064.stdout:1472 write 0xeba96c4 thru 0xebb1c72 (0x85af bytes) 2024-04-24T00:34:42.179 INFO:teuthology.orchestra.run.smithi064.stdout:1473 punch from 0x2411f0b to 0x241fdb1, (0xdea6 bytes) 2024-04-24T00:34:42.179 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:42.179 INFO:teuthology.orchestra.run.smithi064.stdout:1477 write 0xa23acfa thru 0xa23d53a (0x2841 bytes) 2024-04-24T00:34:42.186 INFO:teuthology.orchestra.run.smithi064.stdout:1478 writesame 0xdf8762f thru 0xdf92107 data_size 0x3(0xaad9 bytes) 2024-04-24T00:34:42.223 INFO:teuthology.orchestra.run.smithi064.stdout:1479 trunc from 0xebb1c73 to 0x932e819 2024-04-24T00:34:42.223 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:42.218+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:42.252 INFO:teuthology.orchestra.run.smithi064.stdout:1480 read 0x2148c55 thru 0x2151844 (0x8bf0 bytes) 2024-04-24T00:34:42.252 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:42.252 INFO:teuthology.orchestra.run.smithi064.stdout:1483 read 0x25727cd thru 0x2575f63 (0x3797 bytes) 2024-04-24T00:34:42.252 INFO:teuthology.orchestra.run.smithi064.stdout:1485 read 0x78765ba thru 0x7883dcc (0xd813 bytes) 2024-04-24T00:34:42.252 INFO:teuthology.orchestra.run.smithi064.stdout:1486 read 0x400757 thru 0x400790 (0x3a bytes) 2024-04-24T00:34:42.253 INFO:teuthology.orchestra.run.smithi064.stdout:1487 writesame 0x8b5c4b thru 0x8bbc47 data_size 0x1(0x5ffd bytes) 2024-04-24T00:34:42.287 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:42.288 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:42.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:42.317 INFO:teuthology.orchestra.run.smithi064.stdout:1488 write 0x4430e37 thru 0x443ccb5 (0xbe7f bytes) 2024-04-24T00:34:42.324 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:42.318+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:42.329 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:42.329 INFO:teuthology.orchestra.run.smithi064.stdout:1490 punch from 0x203dec9 to 0x204ab9e, (0xccd5 bytes) 2024-04-24T00:34:42.347 INFO:teuthology.orchestra.run.smithi064.stdout:1491 write 0xb1ea603 thru 0xb1f9134 (0xeb32 bytes) 2024-04-24T00:34:42.359 INFO:teuthology.orchestra.run.smithi064.stdout:1496 read 0x115996e thru 0x115a33e (0x9d1 bytes) 2024-04-24T00:34:42.360 INFO:teuthology.orchestra.run.smithi064.stdout:1499 punch from 0x8464a0c to 0x8473f98, (0xf58c bytes) 2024-04-24T00:34:42.360 INFO:teuthology.orchestra.run.smithi064.stdout:1500 writesame 0xa83b33 thru 0xa866a9 data_size 0x1(0x2b77 bytes) 2024-04-24T00:34:42.371 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:42.371 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:42.371 INFO:teuthology.orchestra.run.smithi064.stdout:1503 write 0x1814ed6 thru 0x181d722 (0x884d bytes) 2024-04-24T00:34:42.375 INFO:teuthology.orchestra.run.smithi064.stdout:1504 writesame 0xa183f40 thru 0xa184101 data_size 0xa(0x1c2 bytes) 2024-04-24T00:34:42.378 INFO:teuthology.orchestra.run.smithi064.stdout:1505 write 0x17d0bf0 thru 0x17e0ab1 (0xfec2 bytes) 2024-04-24T00:34:42.385 INFO:teuthology.orchestra.run.smithi064.stdout:1506 read 0xf0a03c thru 0xf0a94c (0x911 bytes) 2024-04-24T00:34:42.386 INFO:teuthology.orchestra.run.smithi064.stdout:1507 clone 10 order 23 su 65536 sc 4 2024-04-24T00:34:42.424 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:42.420+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:42.436 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:42.436 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:42.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:42.436 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:42.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.436 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:42.436 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.504934292999998, 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:34:42.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:34:42.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:34:42.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:34:42.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4984 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.837590345999999, 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024665092, 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4984 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4e-07 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593755+0000", 2024-04-24T00:34:42.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6829999999999999e-06 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593765+0000", 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7140000000000005e-06 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593773+0000", 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7829999999999999e-06 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609097+0000", 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015324669000000001 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609254+0000", 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015632499999999999 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609918+0000", 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066479400000000002 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.615678+0000", 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057599119999999998 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618123+0000", 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024450909999999999 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618280+0000", 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156706 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618415+0000", 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134475 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.447 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5029 3.f 3:f41ea3ee:::rbd_data.115ed6c33904.0000000000000006:head [zero 262144~262144,zero 786432~65536,zero 917504~39917,write 957421~8409 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.822138981000002, 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024904274000000001, 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5029 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609208+0000", 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8959999999999997e-06 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609285+0000", 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6738999999999998e-05 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609308+0000", 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3896999999999998e-05 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609323+0000", 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4543e-05 2024-04-24T00:34:42.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622194+0000", 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012870985 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622559+0000", 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036503000000000002 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.623244+0000", 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068464100000000005 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.629548+0000", 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063044329999999999 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.633948+0000", 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044003250000000001 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634037+0000", 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9109000000000004e-05 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634105+0000", 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7675999999999994e-05 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5306 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.746003381999998, 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025510631999999998, 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.453 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5306 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:42.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685337+0000", 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685342+0000", 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4560000000000002e-06 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685352+0000", 2024-04-24T00:34:42.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840000000000006e-06 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685359+0000", 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7100000000000007e-06 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696262+0000", 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010902604 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696476+0000", 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021439300000000001 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.697460+0000", 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00098361900000000003 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.706972+0000", 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095118609999999999 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710681+0000", 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037093619999999999 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710762+0000", 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0374e-05 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710847+0000", 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5478999999999999e-05 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.458 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10360 3.11 3:8fda792a:::rbd_data.115e380d621c.0000000000000177:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.979949414, 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042632046999999999, 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10360 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.459 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451393+0000", 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5600000000000001e-06 2024-04-24T00:34:42.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451406+0000", 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2984e-05 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451445+0000", 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9045999999999999e-05 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451461+0000", 2024-04-24T00:34:42.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5786000000000001e-05 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451608+0000", 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014694900000000001 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451914+0000", 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306657 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:42.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002601744 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493753+0000", 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039236409 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493841+0000", 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8448000000000004e-05 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493931+0000", 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0198999999999995e-05 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494023+0000", 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1265000000000001e-05 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10360 3.11 e40/21 ondisk, result = 0)", 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.969084121000002, 2024-04-24T00:34:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031780810999999999, 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462258+0000", 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2539999999999999e-06 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462266+0000", 2024-04-24T00:34:42.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8029999999999997e-06 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462294+0000", 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8637e-05 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462314+0000", 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493814+0000", 2024-04-24T00:34:42.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031500133 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493831+0000", 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7357e-05 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494037+0000", 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020520400000000001 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.572968605, 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.571719857000001, 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.474 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:34:42.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10590 3.13 e41/21 ondisk, result = 0)", 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.560474273000001, 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042592342999999998, 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:42.480 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870868+0000", 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1730000000000002e-06 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870876+0000", 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2069999999999995e-06 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870900+0000", 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.404e-05 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870914+0000", 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.366e-05 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913167+0000", 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042252855999999998 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913184+0000", 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7779e-05 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913458+0000", 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027362799999999999 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.483 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10592 3.16 e41/21 ondisk, result = 0)", 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.556057676999998, 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044174204000000002, 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875284+0000", 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1050000000000002e-06 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875292+0000", 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3200000000000002e-06 2024-04-24T00:34:42.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875319+0000", 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7382999999999999e-05 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875334+0000", 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4786e-05 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919179+0000", 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043845151999999998 2024-04-24T00:34:42.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919195+0000", 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5846999999999999e-05 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919456+0000", 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026161100000000003 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.487 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.414095753000002, 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:34:42.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:34:42.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.400601210000001, 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:42.494 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21553 3.7 3:e2dbd82a:::rbd_data.115eeb9272a5.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.335118104999999, 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029891540000000001, 2024-04-24T00:34:42.498 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21553 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096223+0000", 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3570000000000001e-06 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096230+0000", 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4080000000000003e-06 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096244+0000", 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4236999999999999e-05 2024-04-24T00:34:42.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096251+0000", 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6569999999999997e-06 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096335+0000", 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3997000000000002e-05 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096473+0000", 2024-04-24T00:34:42.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013718199999999999 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.097939+0000", 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014665570000000001 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125952+0000", 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028013198 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125977+0000", 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4613999999999998e-05 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126035+0000", 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7966e-05 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126113+0000", 2024-04-24T00:34:42.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8367000000000004e-05 2024-04-24T00:34:42.506 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.506 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.506 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.138820095, 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.507 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.512 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21613 3.7 e52/44 ondisk, result = 0)", 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.130986156000001, 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044307553999999999, 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:42.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300355+0000", 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4610000000000001e-06 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300362+0000", 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4930000000000003e-06 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300382+0000", 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9887000000000001e-05 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300394+0000", 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2799000000000001e-05 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344558+0000", 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044164017999999999 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344589+0000", 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.027e-05 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344661+0000", 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2626000000000006e-05 2024-04-24T00:34:42.516 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.8852339520000001, 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.517 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:34:42.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:42.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.522 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.8575600879999996, 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:34:42.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.363537268, 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.527 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:34:42.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.3545284340000001, 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:34:42.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30153 3.11 3:8dec0a39:::rbd_data.115e8210e51a.0000000000000015:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.274346046, 2024-04-24T00:34:42.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026293440000000001, 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30153 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:42.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156998+0000", 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7270000000000001e-06 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157022+0000", 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3958999999999999e-05 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157065+0000", 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3643000000000002e-05 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157090+0000", 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.446e-05 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157200+0000", 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110406 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:42.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157303+0000", 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102545 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159693+0000", 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023907830000000001 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183138+0000", 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023444415999999999 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:42.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183198+0000", 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0331999999999998e-05 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183245+0000", 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7154000000000003e-05 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183287+0000", 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2014999999999997e-05 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:42.541 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:42.543 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:42.540+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:42.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:42.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:34:42.644 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:42.640+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:42.689 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:42.689 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:42.689 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:42.689 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:42.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:42.753 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:42.750+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:42.845 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:42.845 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:42.845 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:42.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:42.845 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:42.846 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:42.840+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:42.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:42.947 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:42.944+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:43.024 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:43.024 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:43.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:43.024 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2019 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.488943548000002, 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014241181, 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2019 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:43.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530779+0000", 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530780+0000", 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.686e-06 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.507e-06 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530790+0000", 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5240000000000002e-06 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544487+0000", 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013696612 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544570+0000", 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2704e-05 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545020+0000", 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044978100000000001 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2020 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.488924963000002, 2024-04-24T00:34:43.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014215145, 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2020 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:43.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9499999999999998e-07 2024-04-24T00:34:43.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530799+0000", 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.013e-06 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4289999999999998e-06 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530810+0000", 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4719999999999997e-06 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544812+0000", 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01400174 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544855+0000", 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3217999999999999e-05 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545012+0000", 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000156878 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10361 3.1f 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.566947914, 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032855711000000003, 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10361 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452776+0000", 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.34e-06 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452789+0000", 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2821e-05 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452813+0000", 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3903000000000001e-05 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452825+0000", 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1904e-05 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452918+0000", 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2742000000000001e-05 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453066+0000", 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014841100000000001 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00144988 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484060+0000", 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029543894000000001 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485505+0000", 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014454610000000001 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485582+0000", 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6792000000000006e-05 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485630+0000", 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7562999999999998e-05 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10362 3.17 3:ea0e3a17:::rbd_data.115e380d621c.0000000000000179:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.565819082000001, 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031846629000000001, 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10362 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:43.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453904+0000", 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.23e-07 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453909+0000", 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7989999999999999e-06 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453922+0000", 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2911999999999999e-05 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453930+0000", 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1430000000000001e-06 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454001+0000", 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1239e-05 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454185+0000", 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018420099999999999 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456031+0000", 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018460180000000001 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485548+0000", 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029516936000000001 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485636+0000", 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8568999999999999e-05 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485693+0000", 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6307000000000001e-05 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485749+0000", 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6681999999999997e-05 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10361 3.1f e40/21 ondisk, result = 0)", 2024-04-24T00:34:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.561380553999999, 2024-04-24T00:34:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025729842999999999, 2024-04-24T00:34:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458343+0000", 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.189e-06 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458348+0000", 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5130000000000004e-06 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458364+0000", 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6235e-05 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458374+0000", 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4220000000000006e-06 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484036+0000", 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025662124000000001 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484049+0000", 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3382e-05 2024-04-24T00:34:43.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484071+0000", 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1977999999999999e-05 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10362 3.17 e40/21 ondisk, result = 0)", 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.558302971, 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024141582000000002, 2024-04-24T00:34:43.050 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461421+0000", 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8339999999999999e-06 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461428+0000", 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4939999999999997e-06 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461455+0000", 2024-04-24T00:34:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6571999999999999e-05 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461468+0000", 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3243e-05 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485528+0000", 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024059515 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485539+0000", 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0933e-05 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485561+0000", 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1991e-05 2024-04-24T00:34:43.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:20360 3.1b 3:de2cd4b6:::rbd_data.115ee4742374.0000000000000022:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.522869218, 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100050336, 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 20360 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:43.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496855+0000", 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6060000000000001e-06 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496866+0000", 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0308e-05 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496891+0000", 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5717000000000001e-05 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496904+0000", 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2852e-05 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.497012+0000", 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107873 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.497126+0000", 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114431 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.503011+0000", 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058843769999999997 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596769+0000", 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093757937 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596790+0000", 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1590000000000002e-05 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596851+0000", 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1042000000000003e-05 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596903+0000", 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1603e-05 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:20360 3.1b e48/21 ondisk, result = 0)", 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.505755142000002, 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082812384000000003, 2024-04-24T00:34:43.061 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513968+0000", 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3689999999999999e-06 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513973+0000", 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.673e-06 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513990+0000", 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6806000000000001e-05 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.514003+0000", 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2924e-05 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596727+0000", 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082724848000000004 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596738+0000", 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0501e-05 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596779+0000", 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1263000000000001e-05 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.065 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:20564 3.17 3:ec244480:::rbd_data.115ee4742374.000000000000000d:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.001490678, 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081361157000000003, 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 20564 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018234+0000", 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.43e-06 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018249+0000", 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4892000000000001e-05 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018285+0000", 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6164999999999997e-05 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018301+0000", 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5860000000000001e-05 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018472+0000", 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171602 2024-04-24T00:34:43.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018753+0000", 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000280611 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.022778+0000", 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004025303 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099276+0000", 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076498022999999998 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099401+0000", 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012506299999999999 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099503+0000", 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101409 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099592+0000", 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9798999999999999e-05 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:20564 3.17 e49/21 ondisk, result = 0)", 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.984827194000001, 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064713039, 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034896+0000", 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.53e-06 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034904+0000", 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2230000000000007e-06 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034935+0000", 2024-04-24T00:34:43.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0278e-05 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034949+0000", 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.403e-05 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099371+0000", 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064422222000000001 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099390+0000", 2024-04-24T00:34:43.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8943000000000001e-05 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099608+0000", 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217813 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21501 3.b 3:d2c9feba:::rbd_data.115eeb9272a5.000000000000001b:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.291108426000001, 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.140289634, 2024-04-24T00:34:43.076 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21501 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728616+0000", 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3020000000000002e-06 2024-04-24T00:34:43.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728626+0000", 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0045e-05 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728648+0000", 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1933000000000002e-05 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728657+0000", 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6129999999999993e-06 2024-04-24T00:34:43.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728726+0000", 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8979000000000005e-05 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728846+0000", 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012003 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.733780+0000", 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004933242 2024-04-24T00:34:43.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868807+0000", 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13502688400000001 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868823+0000", 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6237999999999999e-05 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868863+0000", 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0074999999999998e-05 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868903+0000", 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0293000000000002e-05 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21501 3.b e52/44 ondisk, result = 0)", 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:43.082 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.270496255000001, 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11958963, 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:43.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749227+0000", 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.663e-06 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749235+0000", 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7470000000000005e-06 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749256+0000", 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1243999999999999e-05 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749268+0000", 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1968e-05 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868685+0000", 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119416835 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868694+0000", 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1139999999999995e-06 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868815+0000", 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012105899999999999 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.086 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21529 3.1b 3:ddb27a62:::rbd_data.115eeb9272a5.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.078769664999999, 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04919722, 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21529 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940955+0000", 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7310000000000002e-06 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940971+0000", 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6229999999999999e-05 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940999+0000", 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7404000000000001e-05 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941012+0000", 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3315e-05 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941093+0000", 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0828999999999995e-05 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941226+0000", 2024-04-24T00:34:43.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133189 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.945656+0000", 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044297529999999998 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989931+0000", 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044275675 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990003+0000", 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2053999999999998e-05 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990084+0000", 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0507999999999996e-05 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990149+0000", 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5531999999999995e-05 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21529 3.1b e52/21 ondisk, result = 0)", 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.059770019999998, 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030208611, 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959953+0000", 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2720000000000001e-06 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959960+0000", 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4690000000000002e-06 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959979+0000", 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9188999999999999e-05 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959992+0000", 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2833000000000001e-05 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989981+0000", 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029988982000000001 2024-04-24T00:34:43.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989994+0000", 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3603e-05 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990160+0000", 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016626299999999999 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21548 3.6 3:631b2dba:::rbd_data.115eeb9272a5.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:43.096 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.977312246, 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035743242000000001, 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21548 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042411+0000", 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5700000000000001e-07 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042415+0000", 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5839999999999996e-06 2024-04-24T00:34:43.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042427+0000", 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1504e-05 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042432+0000", 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4319999999999998e-06 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042481+0000", 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8557999999999998e-05 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042573+0000", 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2349999999999995e-05 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.045702+0000", 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003128858 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.077952+0000", 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032250612999999997 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078015+0000", 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2855999999999996e-05 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078092+0000", 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6922000000000007e-05 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078153+0000", 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0708000000000002e-05 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21548 3.6 e52/21 ondisk, result = 0)", 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:43.103 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.963564967, 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022006369000000001, 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056158+0000", 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056163+0000", 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8300000000000003e-06 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056180+0000", 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6575999999999999e-05 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056188+0000", 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6219999999999998e-06 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.077995+0000", 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021806968999999999 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078007+0000", 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1771e-05 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078163+0000", 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015619900000000001 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22146 3.18 3:1e672920:::rbd_data.115e6a380959.0000000000000004:head [zero 0~262144,write 262144~262144 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.212865872, 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026558528000000001, 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.107 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22146 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806857+0000", 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.384e-06 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807101+0000", 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024388599999999999 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807118+0000", 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6364000000000001e-05 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807127+0000", 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9469999999999992e-06 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807208+0000", 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0982999999999994e-05 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807431+0000", 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022303899999999999 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.808060+0000", 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062940900000000004 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832854+0000", 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024794415 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833269+0000", 2024-04-24T00:34:43.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041483500000000001 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833342+0000", 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2755000000000004e-05 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833415+0000", 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2510999999999997e-05 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22147 3.18 3:1b6fdb14:::rbd_data.115e6a380959.0000000000000002:head [write 0~482922 [fadvise_sequential+fadvise_nocache],zero 482922~97123,write 580045~47154 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.212566958, 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028316734999999999, 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22147 2024-04-24T00:34:43.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.115 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:43.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:43.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807156+0000", 2024-04-24T00:34:43.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6499999999999998e-07 2024-04-24T00:34:43.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807545+0000", 2024-04-24T00:34:43.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038943700000000002 2024-04-24T00:34:43.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807561+0000", 2024-04-24T00:34:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5724999999999999e-05 2024-04-24T00:34:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807570+0000", 2024-04-24T00:34:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1679999999999993e-06 2024-04-24T00:34:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.820773+0000", 2024-04-24T00:34:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013202667 2024-04-24T00:34:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.820953+0000", 2024-04-24T00:34:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018013799999999999 2024-04-24T00:34:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.821279+0000", 2024-04-24T00:34:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000326244 2024-04-24T00:34:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833920+0000", 2024-04-24T00:34:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012641316999999999 2024-04-24T00:34:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835320+0000", 2024-04-24T00:34:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013997219999999999 2024-04-24T00:34:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835412+0000", 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2283999999999999e-05 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835472+0000", 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9268000000000002e-05 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22148 3.b 3:d51ed8a7:::rbd_data.115e6a380959.0000000000000005:head [write 0~524288 [fadvise_sequential+fadvise_nocache],zero 1048576~524288,zero 5242880~89944,write 5332824~63306 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.209971335000001, 2024-04-24T00:34:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01907449, 2024-04-24T00:34:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22148 2024-04-24T00:34:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809752+0000", 2024-04-24T00:34:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.764e-06 2024-04-24T00:34:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810284+0000", 2024-04-24T00:34:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053170400000000003 2024-04-24T00:34:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810302+0000", 2024-04-24T00:34:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7723e-05 2024-04-24T00:34:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810313+0000", 2024-04-24T00:34:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1486e-05 2024-04-24T00:34:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810413+0000", 2024-04-24T00:34:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9906000000000002e-05 2024-04-24T00:34:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810646+0000", 2024-04-24T00:34:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023295499999999999 2024-04-24T00:34:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810942+0000", 2024-04-24T00:34:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000296218 2024-04-24T00:34:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.823674+0000", 2024-04-24T00:34:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012731732000000001 2024-04-24T00:34:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828700+0000", 2024-04-24T00:34:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050263729999999998 2024-04-24T00:34:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828771+0000", 2024-04-24T00:34:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1027999999999998e-05 2024-04-24T00:34:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828825+0000", 2024-04-24T00:34:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600999999999997e-05 2024-04-24T00:34:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:22146 3.18 e58/44 ondisk, result = 0)", 2024-04-24T00:34:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.206177361, 2024-04-24T00:34:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01931886, 2024-04-24T00:34:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813546+0000", 2024-04-24T00:34:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-04-24T00:34:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813553+0000", 2024-04-24T00:34:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1369999999999996e-06 2024-04-24T00:34:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813574+0000", 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1185000000000001e-05 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813584+0000", 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0297e-05 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832839+0000", 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019254335000000001 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832848+0000", 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9609999999999994e-06 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832863+0000", 2024-04-24T00:34:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5722e-05 2024-04-24T00:34:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:43.147 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:43.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:43.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:43.149 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:43.142+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:43.249 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:43.244+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:43.333 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:43.333 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:43.333 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:43.333 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:43.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:43.350 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:43.344+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:43.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:43.375 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:43.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:43.401 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:43.401 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:43.420 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:43.420 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:43.440 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:43.440 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:43.450 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:43.446+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:43.461 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:43.461 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:43.475 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:43.475 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:43.483 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:43.484 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:43.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:43.506 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:43.551 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:43.548+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:34:43.630 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:43.630 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:43.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:43.693 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:43.690+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:43.752 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:43.748+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:43.852 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:43.853 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:43.853 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:43.848+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:43.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:43.948 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:43.948 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:43.948 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T00:34:43.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T00:34:43.993 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:43.990+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:43.998 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:43.998 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:44.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:44.057 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:44.054+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:44.156 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:44.152+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:44.178 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:44.178 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:44.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:44.249 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:34:44.249 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:44.249 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 68, 2024-04-24T00:34:44.249 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:34:44.249 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-04-24T00:33:24.857711+0000", 2024-04-24T00:34:44.249 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-04-24T00:34:43.578797+0000", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-04-24T00:33:37.880801+0000", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-04-24T00:34:33.763713+0000", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5806080, 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 4, 2024-04-24T00:34:44.250 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 3, 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:39.734814+0000", 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:34:44.251 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-04-24T00:34:44.252 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "17", 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:34:44.253 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:44.254 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:44.256 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 8, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 8, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.25, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:43.786311+0000", 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:34:44.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-04-24T00:34:44.258 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "20", 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:44.259 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:44.261 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 1.9900000095367432, 2024-04-24T00:34:44.262 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 1.9900000095367432, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.75, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.75, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "pool_client.0", 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:51.288702+0000", 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T00:34:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:44.264 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "68", 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 22, 2024-04-24T00:34:44.265 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 68, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:44.267 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.2400000095367432, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.2400000095367432, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:34:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1c80b3f4-99ee-4725-90a8-920116161bf5", 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:44.269 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 53, 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6817", 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6818", 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6819", 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6822", 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.272 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6823", 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6820", 2024-04-24T00:34:44.273 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6821", 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6817/3307242121", 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6819/3307242121", 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6823/3307242121", 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6821/3307242121", 2024-04-24T00:34:44.274 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "20e1331a-f8a2-48fd-b2b8-b7a92f21d78e", 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:44.275 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 59, 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6824", 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6825", 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:44.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6826", 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6827", 2024-04-24T00:34:44.277 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:44.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.278 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.278 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:44.278 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.279 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6830", 2024-04-24T00:34:44.279 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:44.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.279 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.279 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6831", 2024-04-24T00:34:44.280 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:44.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.280 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.280 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:44.280 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.281 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6828", 2024-04-24T00:34:44.281 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:44.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.281 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6829", 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6825/637720411", 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6827/637720411", 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6831/637720411", 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6829/637720411", 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a51557a0-9634-4a20-82b0-b54cdd36b166", 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 55, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:44.283 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/820024667", 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6811/820024667", 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6815/820024667", 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6813/820024667", 2024-04-24T00:34:44.287 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "aafffa79-f532-4441-b1d6-f6441c5eebf8", 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.3392791748046875, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:44.288 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 55, 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:44.290 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.291 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/1974534323", 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6803/1974534323", 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1974534323", 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1974534323", 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "78f05556-598e-4be2-b682-e3f58863eb82", 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:44.293 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 59, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6816", 2024-04-24T00:34:44.294 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6818", 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6820", 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6821", 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.296 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6826", 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6828", 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6823", 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:44.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6824", 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6818/1642543604", 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6821/1642543604", 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6828/1642543604", 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6824/1642543604", 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:44.298 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5", 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 59, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:44.299 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6808", 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6809", 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6810", 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6811", 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6814", 2024-04-24T00:34:44.301 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6815", 2024-04-24T00:34:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6812", 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6813", 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.303 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6809/172726194", 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6811/172726194", 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6815/172726194", 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6813/172726194", 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "6dad12bf-94de-482a-a402-49e67d5e5dc7", 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:44.304 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 59, 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6800", 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6801", 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6802", 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6803", 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6806", 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6807", 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6804", 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6805", 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6801/2195746987", 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6803/2195746987", 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2195746987", 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6805/2195746987", 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.309 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e6734cfe-3c24-4ddd-8ef0-5ed450efe772", 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 62, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6817", 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6819", 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6822", 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6825", 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6830", 2024-04-24T00:34:44.312 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6831", 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:44.313 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6827", 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6829", 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6819/954791674", 2024-04-24T00:34:44.314 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6825/954791674", 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6831/954791674", 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6829/954791674", 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:44.315 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.509233+0000", 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.465696+0000", 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.405177+0000", 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:44.317 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.269202+0000", 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:44.318 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.359180+0000", 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.351834+0000", 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:44.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.490127+0000", 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:34:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.484842+0000", 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.1b", 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:34:44.321 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-04-24T00:34:44.322 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [ 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 18, 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:44.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 20, 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-04-24T00:34:44.324 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-04-24T00:34:44.325 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-04-24T00:34:44.325 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-04-24T00:34:44.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:44.325 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:44.327 INFO:tasks.thrashosds.thrasher:j is {'epoch': 68, 'fsid': '6a70d87d-4453-47be-af2a-5590f9851254', 'created': '2024-04-24T00:33:24.857711+0000', 'modified': '2024-04-24T00:34:43.578797+0000', 'last_up_change': '2024-04-24T00:33:37.880801+0000', 'last_in_change': '2024-04-24T00:34:33.763713+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T00:33:39.734814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T00:33:43.786311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-24T00:33:51.288702+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 22, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '1c80b3f4-99ee-4725-90a8-920116161bf5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 3307242121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 3307242121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 3307242121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 3307242121}]}, 'public_addr': '172.21.15.64:6817/3307242121', 'cluster_addr': '172.21.15.64:6819/3307242121', 'heartbeat_back_addr': '172.21.15.64:6823/3307242121', 'heartbeat_front_addr': '172.21.15.64:6821/3307242121', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 59, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 637720411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 637720411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 637720411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 637720411}]}, 'public_addr': '172.21.15.64:6825/637720411', 'cluster_addr': '172.21.15.64:6827/637720411', 'heartbeat_back_addr': '172.21.15.64:6831/637720411', 'heartbeat_front_addr': '172.21.15.64:6829/637720411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a51557a0-9634-4a20-82b0-b54cdd36b166', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 820024667}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 820024667}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 820024667}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 820024667}]}, 'public_addr': '172.21.15.64:6809/820024667', 'cluster_addr': '172.21.15.64:6811/820024667', 'heartbeat_back_addr': '172.21.15.64:6815/820024667', 'heartbeat_front_addr': '172.21.15.64:6813/820024667', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aafffa79-f532-4441-b1d6-f6441c5eebf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3392791748046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1974534323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1974534323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1974534323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1974534323}]}, 'public_addr': '172.21.15.64:6801/1974534323', 'cluster_addr': '172.21.15.64:6803/1974534323', 'heartbeat_back_addr': '172.21.15.64:6807/1974534323', 'heartbeat_front_addr': '172.21.15.64:6805/1974534323', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '78f05556-598e-4be2-b682-e3f58863eb82', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 59, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6816', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6818', 'nonce': 1642543604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6820', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6821', 'nonce': 1642543604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6826', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6828', 'nonce': 1642543604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6823', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6824', 'nonce': 1642543604}]}, 'public_addr': '172.21.15.167:6818/1642543604', 'cluster_addr': '172.21.15.167:6821/1642543604', 'heartbeat_back_addr': '172.21.15.167:6828/1642543604', 'heartbeat_front_addr': '172.21.15.167:6824/1642543604', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 59, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6808', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 172726194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6810', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 172726194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6814', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 172726194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6812', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 172726194}]}, 'public_addr': '172.21.15.167:6809/172726194', 'cluster_addr': '172.21.15.167:6811/172726194', 'heartbeat_back_addr': '172.21.15.167:6815/172726194', 'heartbeat_front_addr': '172.21.15.167:6813/172726194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6dad12bf-94de-482a-a402-49e67d5e5dc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 59, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6800', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2195746987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6802', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2195746987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6806', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2195746987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6804', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2195746987}]}, 'public_addr': '172.21.15.167:6801/2195746987', 'cluster_addr': '172.21.15.167:6803/2195746987', 'heartbeat_back_addr': '172.21.15.167:6807/2195746987', 'heartbeat_front_addr': '172.21.15.167:6805/2195746987', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e6734cfe-3c24-4ddd-8ef0-5ed450efe772', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6817', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6819', 'nonce': 954791674}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6822', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6825', 'nonce': 954791674}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6830', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6831', 'nonce': 954791674}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6827', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6829', 'nonce': 954791674}]}, 'public_addr': '172.21.15.167:6819/954791674', 'cluster_addr': '172.21.15.167:6825/954791674', 'heartbeat_back_addr': '172.21.15.167:6831/954791674', 'heartbeat_front_addr': '172.21.15.167:6829/954791674', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.509233+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.465696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.405177+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.269202+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.359180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.351834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.490127+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.484842+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1b', 'osds': [5, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 18, 'length': 1}, {'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T00:34:44.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:34:44.328 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:44.328 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:44.328 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:44.322+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:44.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:44.427 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:44.424+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:44.520 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:44.521 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:44.528 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:44.524+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:44.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:44.629 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:44.626+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:44.667 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:44.667 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:44.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:44.698 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:34:44.699 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:34:44.719 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-24T00:34:44.594768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446150600,"num_objects":260,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":31528,"num_read_kb":182443,"num_write":4768,"num_write_kb":142517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":15117802,"num_keys_recovered":100,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3657,"ondisk_log_size":3657,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":247460,"kb_used_data":16472,"kb_used_omap":10,"kb_used_meta":230645,"kb_avail":660355420,"statfs":{"total":676457349120,"available":676203950080,"internally_reserved":0,"allocated":16867328,"data_stored":74039164,"data_compressed":3663206,"data_compressed_allocated":8134656,"data_compressed_original":69867436,"omap_allocated":11157,"internal_metadata":236180587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33497088,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":10095,"num_read_kb":110103,"num_write":1056,"num_write_kb":84369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.933178"},"pg_stats":[{"pgid":"3.1f","version":"66'116","reported_seq":1902,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.685655+0000","last_change":"2024-04-24T00:34:40.601333+0000","last_active":"2024-04-24T00:34:43.685655+0000","last_peered":"2024-04-24T00:34:43.685655+0000","last_clean":"2024-04-24T00:34:43.685655+0000","last_became_active":"2024-04-24T00:34:34.562029+0000","last_became_peered":"2024-04-24T00:34:34.562029+0000","last_unstale":"2024-04-24T00:34:43.685655+0000","last_undegraded":"2024-04-24T00:34:43.685655+0000","last_fullsized":"2024-04-24T00:34:43.685655+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:39:04.943950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010792219999999999,"stat_sum":{"num_bytes":31612234,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1839,"num_read_kb":16942,"num_write":123,"num_write_kb":6630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":107847,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1e","version":"68'130","reported_seq":874,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.608327+0000","last_change":"2024-04-24T00:34:40.597398+0000","last_active":"2024-04-24T00:34:43.608327+0000","last_peered":"2024-04-24T00:34:43.608327+0000","last_clean":"2024-04-24T00:34:43.608327+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:34:43.608327+0000","last_undegraded":"2024-04-24T00:34:43.608327+0000","last_fullsized":"2024-04-24T00:34:43.608327+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:02:33.312341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017168000000000001,"stat_sum":{"num_bytes":13473940,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1084,"num_read_kb":3327,"num_write":155,"num_write_kb":4418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1d","version":"66'88","reported_seq":841,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.688974+0000","last_change":"2024-04-24T00:34:40.600852+0000","last_active":"2024-04-24T00:34:43.688974+0000","last_peered":"2024-04-24T00:34:43.688974+0000","last_clean":"2024-04-24T00:34:43.688974+0000","last_became_active":"2024-04-24T00:34:28.726196+0000","last_became_peered":"2024-04-24T00:34:28.726196+0000","last_unstale":"2024-04-24T00:34:43.688974+0000","last_undegraded":"2024-04-24T00:34:43.688974+0000","last_fullsized":"2024-04-24T00:34:43.688974+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:47:49.179962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082770000000000001,"stat_sum":{"num_bytes":11558194,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1713,"num_read_kb":11912,"num_write":92,"num_write_kb":5648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":458933,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1c","version":"68'178","reported_seq":2026,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.594979+0000","last_change":"2024-04-24T00:34:40.600359+0000","last_active":"2024-04-24T00:34:43.594979+0000","last_peered":"2024-04-24T00:34:43.594979+0000","last_clean":"2024-04-24T00:34:43.594979+0000","last_became_active":"2024-04-24T00:34:31.747052+0000","last_became_peered":"2024-04-24T00:34:31.747052+0000","last_unstale":"2024-04-24T00:34:43.594979+0000","last_undegraded":"2024-04-24T00:34:43.594979+0000","last_fullsized":"2024-04-24T00:34:43.594979+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:27:26.133631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069460099999999999,"stat_sum":{"num_bytes":11868313,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2241,"num_read_kb":10628,"num_write":226,"num_write_kb":1841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":949407,"num_keys_recovered":27,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1b","version":"67'78","reported_seq":164,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.681819+0000","last_change":"2024-04-24T00:34:40.600092+0000","last_active":"2024-04-24T00:34:43.681819+0000","last_peered":"2024-04-24T00:34:43.681819+0000","last_clean":"2024-04-24T00:34:43.681819+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:34:43.681819+0000","last_undegraded":"2024-04-24T00:34:43.681819+0000","last_fullsized":"2024-04-24T00:34:43.681819+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:30:03.068529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041425400000000002,"stat_sum":{"num_bytes":12701911,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":364,"num_read_kb":4315,"num_write":96,"num_write_kb":5013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1a","version":"66'57","reported_seq":1400,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.689129+0000","last_change":"2024-04-24T00:34:40.580949+0000","last_active":"2024-04-24T00:34:43.689129+0000","last_peered":"2024-04-24T00:34:43.689129+0000","last_clean":"2024-04-24T00:34:43.689129+0000","last_became_active":"2024-04-24T00:33:53.027055+0000","last_became_peered":"2024-04-24T00:33:53.027055+0000","last_unstale":"2024-04-24T00:34:43.689129+0000","last_undegraded":"2024-04-24T00:34:43.689129+0000","last_fullsized":"2024-04-24T00:34:43.689129+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:18:07.610707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086990199999999998,"stat_sum":{"num_bytes":7454824,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1175,"num_read_kb":7872,"num_write":74,"num_write_kb":3802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.19","version":"66'137","reported_seq":1405,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.589456+0000","last_change":"2024-04-24T00:34:40.587799+0000","last_active":"2024-04-24T00:34:43.589456+0000","last_peered":"2024-04-24T00:34:43.589456+0000","last_clean":"2024-04-24T00:34:43.589456+0000","last_became_active":"2024-04-24T00:33:53.017357+0000","last_became_peered":"2024-04-24T00:33:53.017357+0000","last_unstale":"2024-04-24T00:34:43.589456+0000","last_undegraded":"2024-04-24T00:34:43.589456+0000","last_fullsized":"2024-04-24T00:34:43.589456+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:28:23.851715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063878100000000005,"stat_sum":{"num_bytes":20216460,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1398,"num_read_kb":10967,"num_write":180,"num_write_kb":1717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.18","version":"66'41","reported_seq":1058,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.685941+0000","last_change":"2024-04-24T00:34:40.588584+0000","last_active":"2024-04-24T00:34:43.685941+0000","last_peered":"2024-04-24T00:34:43.685941+0000","last_clean":"2024-04-24T00:34:43.685941+0000","last_became_active":"2024-04-24T00:34:34.549969+0000","last_became_peered":"2024-04-24T00:34:34.549969+0000","last_unstale":"2024-04-24T00:34:43.685941+0000","last_undegraded":"2024-04-24T00:34:43.685941+0000","last_fullsized":"2024-04-24T00:34:43.685941+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:25.702803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013359190000000001,"stat_sum":{"num_bytes":15964160,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":994,"num_read_kb":12037,"num_write":33,"num_write_kb":2625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":803724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.17","version":"66'74","reported_seq":386,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.595052+0000","last_change":"2024-04-24T00:34:40.601202+0000","last_active":"2024-04-24T00:34:43.595052+0000","last_peered":"2024-04-24T00:34:43.595052+0000","last_clean":"2024-04-24T00:34:43.595052+0000","last_became_active":"2024-04-24T00:34:34.561802+0000","last_became_peered":"2024-04-24T00:34:34.561802+0000","last_unstale":"2024-04-24T00:34:43.595052+0000","last_undegraded":"2024-04-24T00:34:43.595052+0000","last_fullsized":"2024-04-24T00:34:43.595052+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:22:46.539865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001128644,"stat_sum":{"num_bytes":1371437,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":1746,"num_write":95,"num_write_kb":1622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":934056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.16","version":"66'212","reported_seq":1342,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.595382+0000","last_change":"2024-04-24T00:34:40.601677+0000","last_active":"2024-04-24T00:34:43.595382+0000","last_peered":"2024-04-24T00:34:43.595382+0000","last_clean":"2024-04-24T00:34:43.595382+0000","last_became_active":"2024-04-24T00:34:31.750038+0000","last_became_peered":"2024-04-24T00:34:31.750038+0000","last_unstale":"2024-04-24T00:34:43.595382+0000","last_undegraded":"2024-04-24T00:34:43.595382+0000","last_fullsized":"2024-04-24T00:34:43.595382+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:24:35.091052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016095510000000001,"stat_sum":{"num_bytes":18713992,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2083,"num_read_kb":5055,"num_write":271,"num_write_kb":6442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2409097,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.15","version":"66'148","reported_seq":1351,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.677638+0000","last_change":"2024-04-24T00:34:40.745601+0000","last_active":"2024-04-24T00:34:43.677638+0000","last_peered":"2024-04-24T00:34:43.677638+0000","last_clean":"2024-04-24T00:34:43.677638+0000","last_became_active":"2024-04-24T00:33:53.038882+0000","last_became_peered":"2024-04-24T00:33:53.038882+0000","last_unstale":"2024-04-24T00:34:43.677638+0000","last_undegraded":"2024-04-24T00:34:43.677638+0000","last_fullsized":"2024-04-24T00:34:43.677638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:59.635853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025111560000000001,"stat_sum":{"num_bytes":11011522,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1455,"num_read_kb":7696,"num_write":242,"num_write_kb":3975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.14","version":"66'73","reported_seq":1297,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.691819+0000","last_change":"2024-04-24T00:34:40.581268+0000","last_active":"2024-04-24T00:34:43.691819+0000","last_peered":"2024-04-24T00:34:43.691819+0000","last_clean":"2024-04-24T00:34:43.691819+0000","last_became_active":"2024-04-24T00:33:53.032030+0000","last_became_peered":"2024-04-24T00:33:53.032030+0000","last_unstale":"2024-04-24T00:34:43.691819+0000","last_undegraded":"2024-04-24T00:34:43.691819+0000","last_fullsized":"2024-04-24T00:34:43.691819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:24.215331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098799400000000007,"stat_sum":{"num_bytes":17752632,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1058,"num_read_kb":12585,"num_write":75,"num_write_kb":10676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":107,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.596383+0000","last_change":"2024-04-24T00:33:44.961547+0000","last_active":"2024-04-24T00:34:43.596383+0000","last_peered":"2024-04-24T00:34:43.596383+0000","last_clean":"2024-04-24T00:34:43.596383+0000","last_became_active":"2024-04-24T00:33:44.960683+0000","last_became_peered":"2024-04-24T00:33:44.960683+0000","last_unstale":"2024-04-24T00:34:43.596383+0000","last_undegraded":"2024-04-24T00:34:43.596383+0000","last_fullsized":"2024-04-24T00:34:43.596383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:26.939423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"66'45","reported_seq":567,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.589488+0000","last_change":"2024-04-24T00:34:40.588024+0000","last_active":"2024-04-24T00:34:43.589488+0000","last_peered":"2024-04-24T00:34:43.589488+0000","last_clean":"2024-04-24T00:34:43.589488+0000","last_became_active":"2024-04-24T00:34:34.550739+0000","last_became_peered":"2024-04-24T00:34:34.550739+0000","last_unstale":"2024-04-24T00:34:43.589488+0000","last_undegraded":"2024-04-24T00:34:43.589488+0000","last_fullsized":"2024-04-24T00:34:43.589488+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:02:21.166165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075290000000000003,"stat_sum":{"num_bytes":17541160,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":336,"num_read_kb":2113,"num_write":43,"num_write_kb":5722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1269922,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":108,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.589596+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:34:43.589596+0000","last_peered":"2024-04-24T00:34:43.589596+0000","last_clean":"2024-04-24T00:34:43.589596+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:34:43.589596+0000","last_undegraded":"2024-04-24T00:34:43.589596+0000","last_fullsized":"2024-04-24T00:34:43.589596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'74","reported_seq":637,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.654451+0000","last_change":"2024-04-24T00:34:40.746002+0000","last_active":"2024-04-24T00:34:43.654451+0000","last_peered":"2024-04-24T00:34:43.654451+0000","last_clean":"2024-04-24T00:34:43.654451+0000","last_became_active":"2024-04-24T00:34:17.646170+0000","last_became_peered":"2024-04-24T00:34:17.646170+0000","last_unstale":"2024-04-24T00:34:43.654451+0000","last_undegraded":"2024-04-24T00:34:43.654451+0000","last_fullsized":"2024-04-24T00:34:43.654451+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:44:00.121756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023549130000000001,"stat_sum":{"num_bytes":22717185,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":354,"num_read_kb":4220,"num_write":88,"num_write_kb":5665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":112,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.592923+0000","last_change":"2024-04-24T00:34:34.563158+0000","last_active":"2024-04-24T00:34:43.592923+0000","last_peered":"2024-04-24T00:34:43.592923+0000","last_clean":"2024-04-24T00:34:43.592923+0000","last_became_active":"2024-04-24T00:34:34.561890+0000","last_became_peered":"2024-04-24T00:34:34.561890+0000","last_unstale":"2024-04-24T00:34:43.592923+0000","last_undegraded":"2024-04-24T00:34:43.592923+0000","last_fullsized":"2024-04-24T00:34:43.592923+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:53.346112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'59","reported_seq":826,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.663034+0000","last_change":"2024-04-24T00:34:40.588863+0000","last_active":"2024-04-24T00:34:43.663034+0000","last_peered":"2024-04-24T00:34:43.663034+0000","last_clean":"2024-04-24T00:34:43.663034+0000","last_became_active":"2024-04-24T00:33:53.015167+0000","last_became_peered":"2024-04-24T00:33:53.015167+0000","last_unstale":"2024-04-24T00:34:43.663034+0000","last_undegraded":"2024-04-24T00:34:43.663034+0000","last_fullsized":"2024-04-24T00:34:43.663034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:51:26.480808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001428337,"stat_sum":{"num_bytes":15076433,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":595,"num_read_kb":3813,"num_write":62,"num_write_kb":3656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":113,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.589805+0000","last_change":"2024-04-24T00:34:34.563488+0000","last_active":"2024-04-24T00:34:43.589805+0000","last_peered":"2024-04-24T00:34:43.589805+0000","last_clean":"2024-04-24T00:34:43.589805+0000","last_became_active":"2024-04-24T00:34:34.563082+0000","last_became_peered":"2024-04-24T00:34:34.563082+0000","last_unstale":"2024-04-24T00:34:43.589805+0000","last_undegraded":"2024-04-24T00:34:43.589805+0000","last_fullsized":"2024-04-24T00:34:43.589805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:37.613276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'110","reported_seq":1011,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.668588+0000","last_change":"2024-04-24T00:34:40.600497+0000","last_active":"2024-04-24T00:34:43.668588+0000","last_peered":"2024-04-24T00:34:43.668588+0000","last_clean":"2024-04-24T00:34:43.668588+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:34:43.668588+0000","last_undegraded":"2024-04-24T00:34:43.668588+0000","last_fullsized":"2024-04-24T00:34:43.668588+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:10:01.585843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079482800000000003,"stat_sum":{"num_bytes":27646825,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1117,"num_read_kb":1404,"num_write":102,"num_write_kb":2918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.3","version":"66'78","reported_seq":299,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.591167+0000","last_change":"2024-04-24T00:34:40.581406+0000","last_active":"2024-04-24T00:34:43.591167+0000","last_peered":"2024-04-24T00:34:43.591167+0000","last_clean":"2024-04-24T00:34:43.591167+0000","last_became_active":"2024-04-24T00:34:34.549707+0000","last_became_peered":"2024-04-24T00:34:34.549707+0000","last_unstale":"2024-04-24T00:34:43.591167+0000","last_undegraded":"2024-04-24T00:34:43.591167+0000","last_fullsized":"2024-04-24T00:34:43.591167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:23:47.649669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084125499999999995,"stat_sum":{"num_bytes":720936,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":156,"num_read_kb":179,"num_write":85,"num_write_kb":892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":38,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.595073+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:34:43.595073+0000","last_peered":"2024-04-24T00:34:43.595073+0000","last_clean":"2024-04-24T00:34:43.595073+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:34:43.595073+0000","last_undegraded":"2024-04-24T00:34:43.595073+0000","last_fullsized":"2024-04-24T00:34:43.595073+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"67'76","reported_seq":548,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.589779+0000","last_change":"2024-04-24T00:34:40.589684+0000","last_active":"2024-04-24T00:34:43.589779+0000","last_peered":"2024-04-24T00:34:43.589779+0000","last_clean":"2024-04-24T00:34:43.589779+0000","last_became_active":"2024-04-24T00:33:53.012086+0000","last_became_peered":"2024-04-24T00:33:53.012086+0000","last_unstale":"2024-04-24T00:34:43.589779+0000","last_undegraded":"2024-04-24T00:34:43.589779+0000","last_fullsized":"2024-04-24T00:34:43.589779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:23:19.826969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020063949999999998,"stat_sum":{"num_bytes":2544811,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":738,"num_read_kb":1753,"num_write":81,"num_write_kb":1262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":107,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.592574+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:34:43.592574+0000","last_peered":"2024-04-24T00:34:43.592574+0000","last_clean":"2024-04-24T00:34:43.592574+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:34:43.592574+0000","last_undegraded":"2024-04-24T00:34:43.592574+0000","last_fullsized":"2024-04-24T00:34:43.592574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"66'69","reported_seq":521,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.590227+0000","last_change":"2024-04-24T00:34:40.580682+0000","last_active":"2024-04-24T00:34:43.590227+0000","last_peered":"2024-04-24T00:34:43.590227+0000","last_clean":"2024-04-24T00:34:43.590227+0000","last_became_active":"2024-04-24T00:34:34.552406+0000","last_became_peered":"2024-04-24T00:34:34.552406+0000","last_unstale":"2024-04-24T00:34:43.590227+0000","last_undegraded":"2024-04-24T00:34:43.590227+0000","last_fullsized":"2024-04-24T00:34:43.590227+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:03.815632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067050900000000001,"stat_sum":{"num_bytes":14988679,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":292,"num_read_kb":4330,"num_write":91,"num_write_kb":3530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2075913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":107,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.591639+0000","last_change":"2024-04-24T00:33:44.961018+0000","last_active":"2024-04-24T00:34:43.591639+0000","last_peered":"2024-04-24T00:34:43.591639+0000","last_clean":"2024-04-24T00:34:43.591639+0000","last_became_active":"2024-04-24T00:33:44.959946+0000","last_became_peered":"2024-04-24T00:33:44.959946+0000","last_unstale":"2024-04-24T00:34:43.591639+0000","last_undegraded":"2024-04-24T00:34:43.591639+0000","last_fullsized":"2024-04-24T00:34:43.591639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:12.966651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"67'55","reported_seq":218,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.688750+0000","last_change":"2024-04-24T00:34:40.601555+0000","last_active":"2024-04-24T00:34:43.688750+0000","last_peered":"2024-04-24T00:34:43.688750+0000","last_clean":"2024-04-24T00:34:43.688750+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:34:43.688750+0000","last_undegraded":"2024-04-24T00:34:43.688750+0000","last_fullsized":"2024-04-24T00:34:43.688750+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:51:33.944804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001123598,"stat_sum":{"num_bytes":18112812,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":425,"num_read_kb":2226,"num_write":59,"num_write_kb":4829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":38,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.594627+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:34:43.594627+0000","last_peered":"2024-04-24T00:34:43.594627+0000","last_clean":"2024-04-24T00:34:43.594627+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:34:43.594627+0000","last_undegraded":"2024-04-24T00:34:43.594627+0000","last_fullsized":"2024-04-24T00:34:43.594627+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":276,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.591858+0000","last_change":"2024-04-24T00:34:37.565501+0000","last_active":"2024-04-24T00:34:43.591858+0000","last_peered":"2024-04-24T00:34:43.591858+0000","last_clean":"2024-04-24T00:34:43.591858+0000","last_became_active":"2024-04-24T00:34:37.564408+0000","last_became_peered":"2024-04-24T00:34:37.564408+0000","last_unstale":"2024-04-24T00:34:43.591858+0000","last_undegraded":"2024-04-24T00:34:43.591858+0000","last_fullsized":"2024-04-24T00:34:43.591858+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:24:00.564848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"66'184","reported_seq":673,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.643739+0000","last_change":"2024-04-24T00:34:40.748413+0000","last_active":"2024-04-24T00:34:43.643739+0000","last_peered":"2024-04-24T00:34:43.643739+0000","last_clean":"2024-04-24T00:34:43.643739+0000","last_became_active":"2024-04-24T00:33:53.022355+0000","last_became_peered":"2024-04-24T00:33:53.022355+0000","last_unstale":"2024-04-24T00:34:43.643739+0000","last_undegraded":"2024-04-24T00:34:43.643739+0000","last_fullsized":"2024-04-24T00:34:43.643739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:50.227147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043361700000000002,"stat_sum":{"num_bytes":10187337,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":931,"num_read_kb":3660,"num_write":292,"num_write_kb":6177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.9","version":"66'96","reported_seq":944,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.690687+0000","last_change":"2024-04-24T00:34:40.581104+0000","last_active":"2024-04-24T00:34:43.690687+0000","last_peered":"2024-04-24T00:34:43.690687+0000","last_clean":"2024-04-24T00:34:43.690687+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:34:43.690687+0000","last_undegraded":"2024-04-24T00:34:43.690687+0000","last_fullsized":"2024-04-24T00:34:43.690687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:35:51.084978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096048599999999998,"stat_sum":{"num_bytes":32844512,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":700,"num_read_kb":4710,"num_write":94,"num_write_kb":4405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.a","version":"66'193","reported_seq":583,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.596308+0000","last_change":"2024-04-24T00:34:40.596938+0000","last_active":"2024-04-24T00:34:43.596308+0000","last_peered":"2024-04-24T00:34:43.596308+0000","last_clean":"2024-04-24T00:34:43.596308+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:34:43.596308+0000","last_undegraded":"2024-04-24T00:34:43.596308+0000","last_fullsized":"2024-04-24T00:34:43.596308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:59.723369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013086879999999999,"stat_sum":{"num_bytes":9080869,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1105,"num_read_kb":2546,"num_write":330,"num_write_kb":3990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.b","version":"66'41","reported_seq":547,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.590773+0000","last_change":"2024-04-24T00:34:40.580754+0000","last_active":"2024-04-24T00:34:43.590773+0000","last_peered":"2024-04-24T00:34:43.590773+0000","last_clean":"2024-04-24T00:34:43.590773+0000","last_became_active":"2024-04-24T00:34:34.549802+0000","last_became_peered":"2024-04-24T00:34:34.549802+0000","last_unstale":"2024-04-24T00:34:43.590773+0000","last_undegraded":"2024-04-24T00:34:43.590773+0000","last_fullsized":"2024-04-24T00:34:43.590773+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:32:50.312392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069261099999999999,"stat_sum":{"num_bytes":6218291,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":900,"num_read_kb":6531,"num_write":38,"num_write_kb":1184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":722108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.c","version":"66'181","reported_seq":1740,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.595170+0000","last_change":"2024-04-24T00:34:40.600770+0000","last_active":"2024-04-24T00:34:43.595170+0000","last_peered":"2024-04-24T00:34:43.595170+0000","last_clean":"2024-04-24T00:34:43.595170+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:34:43.595170+0000","last_undegraded":"2024-04-24T00:34:43.595170+0000","last_fullsized":"2024-04-24T00:34:43.595170+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:54:05.174285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085112399999999998,"stat_sum":{"num_bytes":30306847,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1882,"num_read_kb":14225,"num_write":253,"num_write_kb":6339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.d","version":"66'30","reported_seq":277,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.592629+0000","last_change":"2024-04-24T00:34:40.744842+0000","last_active":"2024-04-24T00:34:43.592629+0000","last_peered":"2024-04-24T00:34:43.592629+0000","last_clean":"2024-04-24T00:34:43.592629+0000","last_became_active":"2024-04-24T00:34:28.729203+0000","last_became_peered":"2024-04-24T00:34:28.729203+0000","last_unstale":"2024-04-24T00:34:43.592629+0000","last_undegraded":"2024-04-24T00:34:43.592629+0000","last_fullsized":"2024-04-24T00:34:43.592629+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:57.516102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016199700000000001,"stat_sum":{"num_bytes":1147326,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":58,"num_read_kb":1235,"num_write":36,"num_write_kb":1203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":532930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.e","version":"66'123","reported_seq":574,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.593376+0000","last_change":"2024-04-24T00:34:40.744831+0000","last_active":"2024-04-24T00:34:43.593376+0000","last_peered":"2024-04-24T00:34:43.593376+0000","last_clean":"2024-04-24T00:34:43.593376+0000","last_became_active":"2024-04-24T00:33:53.031663+0000","last_became_peered":"2024-04-24T00:33:53.031663+0000","last_unstale":"2024-04-24T00:34:43.593376+0000","last_undegraded":"2024-04-24T00:34:43.593376+0000","last_fullsized":"2024-04-24T00:34:43.593376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:18:19.572891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001506786,"stat_sum":{"num_bytes":13121882,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":658,"num_read_kb":5408,"num_write":207,"num_write_kb":3362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.f","version":"66'124","reported_seq":1194,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.685613+0000","last_change":"2024-04-24T00:34:40.746518+0000","last_active":"2024-04-24T00:34:43.685613+0000","last_peered":"2024-04-24T00:34:43.685613+0000","last_clean":"2024-04-24T00:34:43.685613+0000","last_became_active":"2024-04-24T00:33:53.029304+0000","last_became_peered":"2024-04-24T00:33:53.029304+0000","last_unstale":"2024-04-24T00:34:43.685613+0000","last_undegraded":"2024-04-24T00:34:43.685613+0000","last_fullsized":"2024-04-24T00:34:43.685613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:04:30.196844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025750600000000001,"stat_sum":{"num_bytes":14871389,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1129,"num_read_kb":6745,"num_write":197,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.10","version":"67'240","reported_seq":1096,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.697670+0000","last_change":"2024-04-24T00:34:40.596865+0000","last_active":"2024-04-24T00:34:43.697670+0000","last_peered":"2024-04-24T00:34:43.697670+0000","last_clean":"2024-04-24T00:34:43.697670+0000","last_became_active":"2024-04-24T00:34:34.557992+0000","last_became_peered":"2024-04-24T00:34:34.557992+0000","last_unstale":"2024-04-24T00:34:43.697670+0000","last_undegraded":"2024-04-24T00:34:43.697670+0000","last_fullsized":"2024-04-24T00:34:43.697670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:35:09.683193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012703059999999999,"stat_sum":{"num_bytes":5679268,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1990,"num_read_kb":3712,"num_write":327,"num_write_kb":3314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":179074,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.11","version":"66'173","reported_seq":755,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.651203+0000","last_change":"2024-04-24T00:34:40.743944+0000","last_active":"2024-04-24T00:34:43.651203+0000","last_peered":"2024-04-24T00:34:43.651203+0000","last_clean":"2024-04-24T00:34:43.651203+0000","last_became_active":"2024-04-24T00:33:53.026709+0000","last_became_peered":"2024-04-24T00:33:53.026709+0000","last_unstale":"2024-04-24T00:34:43.651203+0000","last_undegraded":"2024-04-24T00:34:43.651203+0000","last_fullsized":"2024-04-24T00:34:43.651203+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:31.497064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095214500000000005,"stat_sum":{"num_bytes":16512305,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":829,"num_read_kb":1917,"num_write":226,"num_write_kb":5334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.12","version":"66'216","reported_seq":812,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.591909+0000","last_change":"2024-04-24T00:34:40.745531+0000","last_active":"2024-04-24T00:34:43.591909+0000","last_peered":"2024-04-24T00:34:43.591909+0000","last_clean":"2024-04-24T00:34:43.591909+0000","last_became_active":"2024-04-24T00:34:37.560744+0000","last_became_peered":"2024-04-24T00:34:37.560744+0000","last_unstale":"2024-04-24T00:34:43.591909+0000","last_undegraded":"2024-04-24T00:34:43.591909+0000","last_fullsized":"2024-04-24T00:34:43.591909+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:16:14.509766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021537969999999998,"stat_sum":{"num_bytes":11137655,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1276,"num_read_kb":5000,"num_write":289,"num_write_kb":8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1710307,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.13","version":"67'41","reported_seq":602,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T00:34:43.592139+0000","last_change":"2024-04-24T00:34:40.744236+0000","last_active":"2024-04-24T00:34:43.592139+0000","last_peered":"2024-04-24T00:34:43.592139+0000","last_clean":"2024-04-24T00:34:43.592139+0000","last_became_active":"2024-04-24T00:33:53.033430+0000","last_became_peered":"2024-04-24T00:33:53.033430+0000","last_unstale":"2024-04-24T00:34:43.592139+0000","last_undegraded":"2024-04-24T00:34:43.592139+0000","last_fullsized":"2024-04-24T00:34:43.592139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:45:57.451268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011556660000000001,"stat_sum":{"num_bytes":1404091,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":394,"num_read_kb":1534,"num_write":47,"num_write_kb":822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"14","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":445560232,"num_objects":258,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":31412,"num_read_kb":182343,"num_write":4609,"num_write_kb":140101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":14527434,"num_keys_recovered":100,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12505088,"data_stored":77726537,"data_compressed":4081239,"data_compressed_allocated":8953856,"data_compressed_original":75892652,"omap_allocated":0,"internal_metadata":0},"log_size":3540,"ondisk_log_size":3540,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":19233,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2789376,"data_stored":14335005,"data_compressed":652808,"data_compressed_allocated":1515520,"data_compressed_original":13720950,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1646592,"data_stored":5716595,"data_compressed":286905,"data_compressed_allocated":622592,"data_compressed_original":5230575,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33956,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337884,"statfs":{"total":96636764160,"available":96601993216,"internally_reserved":0,"allocated":2846720,"data_stored":11922944,"data_compressed":666293,"data_compressed_allocated":1404928,"data_compressed_original":11214848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38400,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333440,"statfs":{"total":96636764160,"available":96597442560,"internally_reserved":0,"allocated":3129344,"data_stored":16629058,"data_compressed":671232,"data_compressed_allocated":1679360,"data_compressed_original":15923783,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38656,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333184,"statfs":{"total":96636764160,"available":96597180416,"internally_reserved":0,"allocated":3149824,"data_stored":14880068,"data_compressed":790996,"data_compressed_allocated":1712128,"data_compressed_original":14184448,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1519616,"data_stored":3610340,"data_compressed":281804,"data_compressed_allocated":475136,"data_compressed_original":3125248,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1785856,"data_stored":6945154,"data_compressed":313168,"data_compressed_allocated":724992,"data_compressed_original":6467584,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":55834574866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":7369185,"data_compressed":431142,"data_compressed_allocated":892928,"data_compressed_original":7188480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1003520,"data_stored":6588734,"data_compressed":313168,"data_compressed_allocated":724992,"data_compressed_original":6467584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":724992,"data_stored":3247704,"data_compressed":281804,"data_compressed_allocated":475136,"data_compressed_original":3125248,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2367488,"data_stored":14523648,"data_compressed":790996,"data_compressed_allocated":1712128,"data_compressed_original":14184448,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2351104,"data_stored":16272678,"data_compressed":671232,"data_compressed_allocated":1679360,"data_compressed_original":15923783,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":11566564,"data_compressed":666293,"data_compressed_allocated":1404928,"data_compressed_original":11214848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":4769807,"data_compressed":279920,"data_compressed_allocated":585728,"data_compressed_original":4640751,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1949696,"data_stored":13388217,"data_compressed":646684,"data_compressed_allocated":1478656,"data_compressed_original":13147510,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:34:44.722 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [5, 7] 2024-04-24T00:34:44.722 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.4', '5', '7'] 2024-04-24T00:34:44.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.4 5 7 2024-04-24T00:34:44.761 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:44.758+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:44.764 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-04-24T00:34:44.819 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:44.819 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:44.832 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:44.828+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:44.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:44.934 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:44.932+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:44.992 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:44.992 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:45.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:45.051 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:45.048+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:45.137 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:45.130+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:45.169 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:45.174 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:45.174 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:45.190 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:45.193 INFO:teuthology.orchestra.run.smithi064.stdout:1510 writesame 0xecb187c thru 0xecbf257 data_size 0x92(0xd9dc bytes) 2024-04-24T00:34:45.193 INFO:teuthology.orchestra.run.smithi064.stdout:1511 read 0x8d89488 thru 0x8d8bcd3 (0x284c bytes) 2024-04-24T00:34:45.193 INFO:teuthology.orchestra.run.smithi064.stdout:1512 punch from 0x74436e8 to 0x744d09b, (0x99b3 bytes) 2024-04-24T00:34:45.194 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:45.194 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:45.194 INFO:teuthology.orchestra.run.smithi064.stdout:1520 writesame 0xa90c403 thru 0xa91575d data_size 0x11(0x935b bytes) 2024-04-24T00:34:45.198 INFO:teuthology.orchestra.run.smithi064.stdout:1521 punch from 0x4eb811f to 0x4ebb280, (0x3161 bytes) 2024-04-24T00:34:45.198 INFO:teuthology.orchestra.run.smithi064.stdout:1523 read 0x6fc1307 thru 0x6fcbe94 (0xab8e bytes) 2024-04-24T00:34:45.198 INFO:teuthology.orchestra.run.smithi064.stdout:1524 read 0x6b2d83e thru 0x6b3691e (0x90e1 bytes) 2024-04-24T00:34:45.199 INFO:teuthology.orchestra.run.smithi064.stdout:1526 writesame 0x324af0e thru 0x324f7bd data_size 0x8(0x48b0 bytes) 2024-04-24T00:34:45.205 INFO:teuthology.orchestra.run.smithi064.stdout:1527 write 0x2d111c7 thru 0x2d1d4f2 (0xc32c bytes) 2024-04-24T00:34:45.213 INFO:teuthology.orchestra.run.smithi064.stdout:1528 write 0x27cd75 thru 0x28974e (0xc9da bytes) 2024-04-24T00:34:45.231 INFO:teuthology.orchestra.run.smithi064.stdout:1529 trunc from 0xecbf258 to 0x5b1fefa 2024-04-24T00:34:45.238 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:45.232+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:45.257 INFO:teuthology.orchestra.run.smithi064.stdout:1530 write 0x43c2f18 thru 0x43c943d (0x6526 bytes) 2024-04-24T00:34:45.271 INFO:teuthology.orchestra.run.smithi064.stdout:1532 write 0x7085b97 thru 0x708606b (0x4d5 bytes) 2024-04-24T00:34:45.276 INFO:teuthology.orchestra.run.smithi064.stdout:1533 punch from 0x33c090b to 0x33c37bc, (0x2eb1 bytes) 2024-04-24T00:34:45.277 INFO:teuthology.orchestra.run.smithi064.stdout:1534 writesame 0x2b09358 thru 0x2b17cfe data_size 0x7(0xe9a7 bytes) 2024-04-24T00:34:45.286 INFO:teuthology.orchestra.run.smithi064.stdout:1536 read 0x59d552a thru 0x59dc5b6 (0x708d bytes) 2024-04-24T00:34:45.287 INFO:teuthology.orchestra.run.smithi064.stdout:1538 read 0x499c82a thru 0x499f296 (0x2a6d bytes) 2024-04-24T00:34:45.289 INFO:teuthology.orchestra.run.smithi064.stdout:1540 writesame 0x49ba24e thru 0x49c1b26 data_size 0x1(0x78d9 bytes) 2024-04-24T00:34:45.320 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:45.320 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:45.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:45.377 INFO:teuthology.orchestra.run.smithi064.stdout:1541 punch from 0x41aa50b to 0x41acef2, (0x29e7 bytes) 2024-04-24T00:34:45.377 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:45.383 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:45.380+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:45.391 INFO:teuthology.orchestra.run.smithi064.stdout:1551 trunc from 0x708606c to 0xc45e38a 2024-04-24T00:34:45.415 INFO:teuthology.orchestra.run.smithi064.stdout:1552 writesame 0xc592a37 thru 0xc5a148b data_size 0x1(0xea55 bytes) 2024-04-24T00:34:45.428 INFO:teuthology.orchestra.run.smithi064.stdout:1553 trunc from 0xc5a148c to 0x7a7c060 2024-04-24T00:34:45.438 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:45.434+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:45.449 INFO:teuthology.orchestra.run.smithi064.stdout:1554 trunc from 0x7a7c060 to 0x914b1f8 2024-04-24T00:34:45.463 INFO:teuthology.orchestra.run.smithi064.stdout:1555 read 0x5fecae7 thru 0x5ffa522 (0xda3c bytes) 2024-04-24T00:34:45.464 INFO:teuthology.orchestra.run.smithi064.stdout:1557 write 0x34c247f thru 0x34ce28a (0xbe0c bytes) 2024-04-24T00:34:45.474 INFO:teuthology.orchestra.run.smithi064.stdout:1558 writesame 0x44f847a thru 0x44ff9ff data_size 0xe(0x7586 bytes) 2024-04-24T00:34:45.480 INFO:teuthology.orchestra.run.smithi064.stdout:1559 trunc from 0x914b1f8 to 0x8412203 2024-04-24T00:34:45.495 INFO:teuthology.orchestra.run.smithi064.stdout:1562 read 0x3cc30a9 thru 0x3cd1aff (0xea57 bytes) 2024-04-24T00:34:45.508 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:45.508 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:45.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:45.525 INFO:teuthology.orchestra.run.smithi064.stdout:1563 writesame 0xdd678f1 thru 0xdd6a01f data_size 0x1(0x272f bytes) 2024-04-24T00:34:45.525 INFO:teuthology.orchestra.run.smithi064.stdout:1564 write 0x9a84e22 thru 0x9a8d075 (0x8254 bytes) 2024-04-24T00:34:45.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:45.534 INFO:teuthology.orchestra.run.smithi064.stdout:1565 trunc from 0xdd6a020 to 0xe011992 2024-04-24T00:34:45.539 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:45.534+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:45.549 INFO:teuthology.orchestra.run.smithi064.stdout:1568 trunc from 0xe011992 to 0xb2af518 2024-04-24T00:34:45.558 INFO:teuthology.orchestra.run.smithi064.stdout:1569 read 0x8c6c427 thru 0x8c76ea1 (0xaa7b bytes) 2024-04-24T00:34:45.558 INFO:teuthology.orchestra.run.smithi064.stdout:1570 read 0x8e65ce1 thru 0x8e70a2e (0xad4e bytes) 2024-04-24T00:34:45.558 INFO:teuthology.orchestra.run.smithi064.stdout:1572 punch from 0x9358902 to 0x93611b8, (0x88b6 bytes) 2024-04-24T00:34:45.575 INFO:teuthology.orchestra.run.smithi064.stdout:1573 trunc from 0xb2af518 to 0x2888916 2024-04-24T00:34:45.593 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:45.593 INFO:teuthology.orchestra.run.smithi064.stdout:1576 read 0x1ba1329 thru 0x1ba2267 (0xf3f bytes) 2024-04-24T00:34:45.600 INFO:teuthology.orchestra.run.smithi064.stdout:1577 trunc from 0x2888916 to 0x20496a 2024-04-24T00:34:45.604 INFO:teuthology.orchestra.run.smithi064.stderr:set 2.4 pg_upmap mapping to [5,7] 2024-04-24T00:34:45.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:34:45.630 INFO:teuthology.orchestra.run.smithi064.stdout:1578 read 0x114144 thru 0x114a8d (0x94a bytes) 2024-04-24T00:34:45.637 INFO:teuthology.orchestra.run.smithi064.stdout:1579 trunc from 0x20496a to 0x434787b 2024-04-24T00:34:45.640 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:45.634+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:45.648 INFO:teuthology.orchestra.run.smithi064.stdout:1580 read 0x375d8bc thru 0x375e3ca (0xb0f bytes) 2024-04-24T00:34:45.650 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:45.650 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:45.661 INFO:teuthology.orchestra.run.smithi064.stdout:1581 writesame 0x6e62976 thru 0x6e65c26 data_size 0x1(0x32b1 bytes) 2024-04-24T00:34:45.661 INFO:teuthology.orchestra.run.smithi064.stdout:1583 read 0x425e6e0 thru 0x4269004 (0xa925 bytes) 2024-04-24T00:34:45.661 INFO:teuthology.orchestra.run.smithi064.stdout:1584 writesame 0x1fef17f thru 0x1ffcbc6 data_size 0x7f(0xda48 bytes) 2024-04-24T00:34:45.664 INFO:teuthology.orchestra.run.smithi064.stdout:1585 read 0x3068748 thru 0x3077743 (0xeffc bytes) 2024-04-24T00:34:45.665 INFO:teuthology.orchestra.run.smithi064.stdout:1587 trunc from 0x6e65c27 to 0x6a4f6d8 2024-04-24T00:34:45.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:45.672 INFO:teuthology.orchestra.run.smithi064.stdout:1588 write 0x3d58008 thru 0x3d5d2f6 (0x52ef bytes) 2024-04-24T00:34:45.677 INFO:teuthology.orchestra.run.smithi064.stdout:1589 read 0x296aca7 thru 0x2974488 (0x97e2 bytes) 2024-04-24T00:34:45.677 INFO:teuthology.orchestra.run.smithi064.stdout:1590 writesame 0x3c8ac22 thru 0x3c9693b data_size 0x2f(0xbd1a bytes) 2024-04-24T00:34:45.681 INFO:teuthology.orchestra.run.smithi064.stdout:1591 read 0x346f419 thru 0x34723d8 (0x2fc0 bytes) 2024-04-24T00:34:45.681 INFO:teuthology.orchestra.run.smithi064.stdout:1592 read 0x5eba4c0 thru 0x5ec82b5 (0xddf6 bytes) 2024-04-24T00:34:45.681 INFO:teuthology.orchestra.run.smithi064.stdout:1593 write 0x45abdf8 thru 0x45b2a10 (0x6c19 bytes) 2024-04-24T00:34:45.689 INFO:teuthology.orchestra.run.smithi064.stdout:1595 read 0x946269 thru 0x94bcaf (0x5a47 bytes) 2024-04-24T00:34:45.689 INFO:teuthology.orchestra.run.smithi064.stdout:1597 read 0x52c392d thru 0x52c71d6 (0x38aa bytes) 2024-04-24T00:34:45.690 INFO:teuthology.orchestra.run.smithi064.stdout:1598 read 0x1d1a6c6 thru 0x1d1c3ec (0x1d27 bytes) 2024-04-24T00:34:45.696 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:45.697 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:45.703 INFO:teuthology.orchestra.run.smithi064.stdout:1599 writesame 0xaf9d2e7 thru 0xaf9fe86 data_size 0x10(0x2ba0 bytes) 2024-04-24T00:34:45.710 INFO:teuthology.orchestra.run.smithi064.stdout:1600 write 0xbfa9295 thru 0xbfb6b4b (0xd8b7 bytes) 2024-04-24T00:34:45.714 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:45.715 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:45.715 INFO:teuthology.orchestra.run.smithi064.stdout:1601 write 0x85e1fd6 thru 0x85ecaac (0xaad7 bytes) 2024-04-24T00:34:45.719 INFO:teuthology.orchestra.run.smithi064.stdout:1605 read 0x87e7a0c thru 0x87f382b (0xbe20 bytes) 2024-04-24T00:34:45.729 INFO:teuthology.orchestra.run.smithi064.stdout:1606 writesame 0xd6b8023 thru 0xd6c365d data_size 0x1(0xb63b bytes) 2024-04-24T00:34:45.729 INFO:teuthology.orchestra.run.smithi064.stdout:1607 write 0x5268d8c thru 0x5269cd5 (0xf4a bytes) 2024-04-24T00:34:45.730 INFO:teuthology.orchestra.run.smithi064.stdout:1608 write 0x9793dd1 thru 0x979c5f5 (0x8825 bytes) 2024-04-24T00:34:45.733 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:45.733 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:45.735 INFO:teuthology.orchestra.run.smithi064.stdout:1610 punch from 0x6233a81 to 0x623ac27, (0x71a6 bytes) 2024-04-24T00:34:45.735 INFO:teuthology.orchestra.run.smithi064.stdout:1612 write 0xc8a0ac5 thru 0xc8a7a90 (0x6fcc bytes) 2024-04-24T00:34:45.739 INFO:teuthology.orchestra.run.smithi064.stdout:1614 read 0x318504a thru 0x319193f (0xc8f6 bytes) 2024-04-24T00:34:45.740 INFO:teuthology.orchestra.run.smithi064.stdout:1616 write 0xe907ca thru 0xe95029 (0x4860 bytes) 2024-04-24T00:34:45.741 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:45.736+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:45.742 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:45.742 INFO:teuthology.orchestra.run.smithi064.stdout:1620 write 0x6da3f99 thru 0x6da758f (0x35f7 bytes) 2024-04-24T00:34:45.746 INFO:teuthology.orchestra.run.smithi064.stdout:1621 writesame 0x42c182c thru 0x42c2e52 data_size 0x1(0x1627 bytes) 2024-04-24T00:34:45.754 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:45.754 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:45.761 INFO:teuthology.orchestra.run.smithi064.stdout:1623 trunc from 0xd6c365e to 0x503b8a6 2024-04-24T00:34:45.779 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:45.779 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:45.779 INFO:teuthology.orchestra.run.smithi064.stdout:1624 writesame 0xbe66ea thru 0xbe9222 data_size 0x1(0x2b39 bytes) 2024-04-24T00:34:45.795 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:45.795 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:45.796 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:45.796 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:45.798 INFO:teuthology.orchestra.run.smithi064.stdout:1625 read 0x299ad5a thru 0x29a744b (0xc6f2 bytes) 2024-04-24T00:34:45.799 INFO:teuthology.orchestra.run.smithi064.stdout:1626 punch from 0x28af041 to 0x28b79da, (0x8999 bytes) 2024-04-24T00:34:45.799 INFO:teuthology.orchestra.run.smithi064.stdout:1627 writesame 0x4f6eb92 thru 0x4f76546 data_size 0x1(0x79b5 bytes) 2024-04-24T00:34:45.816 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:45.816 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:45.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:45.833 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:45.834 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:45.842 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:45.838+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:45.881 INFO:teuthology.orchestra.run.smithi064.stdout:1628 writesame 0xe7da65a thru 0xe7dad14 data_size 0x1(0x6bb bytes) 2024-04-24T00:34:45.882 INFO:teuthology.orchestra.run.smithi064.stdout:1629 writesame 0x680a782 thru 0x6817230 data_size 0x1(0xcaaf bytes) 2024-04-24T00:34:45.943 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:45.940+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:45.957 INFO:teuthology.orchestra.run.smithi064.stdout:1630 trunc from 0xe7dad15 to 0x529e874 2024-04-24T00:34:45.991 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:45.991 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:45.992 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:45.992 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:45.992 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:45.999 INFO:teuthology.orchestra.run.smithi064.stdout:1631 write 0x71c579a thru 0x71d2d3a (0xd5a1 bytes) 2024-04-24T00:34:46.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:46.029 INFO:teuthology.orchestra.run.smithi064.stdout:1634 write 0xb1ad79a thru 0xb1b4923 (0x718a bytes) 2024-04-24T00:34:46.036 INFO:teuthology.orchestra.run.smithi064.stdout:1636 writesame 0x46018e4 thru 0x4604fb5 data_size 0x3(0x36d2 bytes) 2024-04-24T00:34:46.040 INFO:teuthology.orchestra.run.smithi064.stdout:1637 punch from 0x31f6b8c to 0x32069a6, (0xfe1a bytes) 2024-04-24T00:34:46.040 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:46.040 INFO:teuthology.orchestra.run.smithi064.stdout:1639 read 0x29702a5 thru 0x297272c (0x2488 bytes) 2024-04-24T00:34:46.040 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:46.041 INFO:teuthology.orchestra.run.smithi064.stdout:1642 writesame 0x650d107 thru 0x651765f data_size 0x1(0xa559 bytes) 2024-04-24T00:34:46.044 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:46.038+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:46.071 INFO:teuthology.orchestra.run.smithi064.stdout:1644 read 0x126b5ed thru 0x12777e3 (0xc1f7 bytes) 2024-04-24T00:34:46.093 INFO:teuthology.orchestra.run.smithi064.stdout:1648 trunc from 0xb1b4924 to 0x2bd7cac 2024-04-24T00:34:46.118 INFO:teuthology.orchestra.run.smithi064.stdout:1649 writesame 0xffdefe thru 0x1004739 data_size 0xe(0x683c bytes) 2024-04-24T00:34:46.129 INFO:teuthology.orchestra.run.smithi064.stdout:1650 trunc from 0x2bd7cac to 0x54eccd5 2024-04-24T00:34:46.142 INFO:teuthology.orchestra.run.smithi064.stdout:1652 read 0x51dc123 thru 0x51debdb (0x2ab9 bytes) 2024-04-24T00:34:46.142 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:46.142 INFO:teuthology.orchestra.run.smithi064.stdout:1654 write 0x92ea16 thru 0x937282 (0x886d bytes) 2024-04-24T00:34:46.146 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:46.140+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:46.165 INFO:teuthology.orchestra.run.smithi064.stdout:1656 writesame 0x6fc9c58 thru 0x6fd4028 data_size 0x7(0xa3d1 bytes) 2024-04-24T00:34:46.165 INFO:teuthology.orchestra.run.smithi064.stdout:1658 writesame 0x3867f9 thru 0x38ebd2 data_size 0x7(0x83da bytes) 2024-04-24T00:34:46.177 INFO:teuthology.orchestra.run.smithi064.stdout:1661 read 0x955ad2 thru 0x95c6fd (0x6c2c bytes) 2024-04-24T00:34:46.178 INFO:teuthology.orchestra.run.smithi064.stdout:1662 writesame 0x4978af7 thru 0x497bb11 data_size 0x5(0x301b bytes) 2024-04-24T00:34:46.190 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:5336 3.1d e36/21 3:b83bad9e:::rbd_data.115ed6c33904.0000000000000086:head v 36'3, mlcod=34'1)", 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.491478231000002, 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010615080000000001, 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694760+0000", 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8080000000000001e-06 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694891+0000", 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013035600000000001 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694916+0000", 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5392e-05 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694930+0000", 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4059999999999999e-05 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695055+0000", 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124499 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695112+0000", 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7853000000000003e-05 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705273+0000", 2024-04-24T00:34:46.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010160626000000001 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705373+0000", 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100486 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10257 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.983834272000003, 2024-04-24T00:34:46.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011690176, 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10257 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202404+0000", 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.897e-06 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202412+0000", 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202430+0000", 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8043e-05 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202442+0000", 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439999999999999e-05 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207283+0000", 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048410600000000003 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207552+0000", 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026864100000000002 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.208755+0000", 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012028189999999999 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.211687+0000", 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029325240000000002 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.213944+0000", 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022567300000000002 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214023+0000", 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8932999999999998e-05 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214093+0000", 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9538999999999999e-05 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10263 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.893398984999997, 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091981449999999996, 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.201 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10263 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292838+0000", 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292839+0000", 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.113e-06 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292846+0000", 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.81e-06 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292862+0000", 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6456e-05 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.292870+0000", 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3699999999999995e-06 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296812+0000", 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039413900000000003 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.296881+0000", 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9426999999999998e-05 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.297166+0000", 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028513300000000002 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301682+0000", 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045159029999999999 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301927+0000", 2024-04-24T00:34:46.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024475499999999998 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.301980+0000", 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2796000000000001e-05 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.302036+0000", 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5992000000000002e-05 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10288 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.853490858000001, 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.009564744, 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10288 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332747+0000", 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:46.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332769+0000", 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1903e-05 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332780+0000", 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1243000000000001e-05 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332789+0000", 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0699999999999996e-06 2024-04-24T00:34:46.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336540+0000", 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037510339999999999 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336760+0000", 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021991099999999999 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.337418+0000", 2024-04-24T00:34:46.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000658127 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.339101+0000", 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016826390000000001 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342150+0000", 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030494850000000002 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342236+0000", 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6212999999999996e-05 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342311+0000", 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4228999999999997e-05 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10365 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.691464639000003, 2024-04-24T00:34:46.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098439340000000004, 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10365 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494773+0000", 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1399999999999995e-07 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494782+0000", 2024-04-24T00:34:46.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3649999999999993e-06 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494794+0000", 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1606e-05 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494804+0000", 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0013999999999999e-05 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498700+0000", 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038961310000000002 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498969+0000", 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000269432 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.499771+0000", 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080119200000000005 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.503502+0000", 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037318300000000002 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504304+0000", 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080115000000000004 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504475+0000", 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017146199999999999 2024-04-24T00:34:46.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504616+0000", 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014083800000000001 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10409 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:46.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.609882438, 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010506112, 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10409 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:46.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576355+0000", 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576359+0000", 2024-04-24T00:34:46.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3229999999999999e-06 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576372+0000", 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576383+0000", 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1192999999999999e-05 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580539+0000", 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041557230000000001 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580813+0000", 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027430400000000001 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.582037+0000", 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001223879 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.584201+0000", 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002164408 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586567+0000", 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023657090000000001 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586731+0000", 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016439800000000001 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586860+0000", 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128835 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10645 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.158878356000002, 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010437706999999999, 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10645 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027359+0000", 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.06e-06 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027365+0000", 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2769999999999996e-06 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027380+0000", 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5537e-05 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027389+0000", 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5299999999999996e-06 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.030838+0000", 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003449232 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031115+0000", 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000277254 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031925+0000", 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080932599999999997 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.035618+0000", 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003693647 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037627+0000", 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020091390000000001 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037720+0000", 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2775000000000005e-05 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037796+0000", 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.593e-05 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.229 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10719 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.007707189000001, 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010945605000000001, 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.230 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10719 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178530+0000", 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1800000000000005e-07 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178535+0000", 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5129999999999998e-06 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178544+0000", 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2359999999999993e-06 2024-04-24T00:34:46.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178550+0000", 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.421e-06 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181685+0000", 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031346989999999999 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181965+0000", 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027960399999999997 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.182619+0000", 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065424799999999996 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.186425+0000", 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038062690000000001 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189312+0000", 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028871809999999999 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189407+0000", 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4345000000000006e-05 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189475+0000", 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8270999999999997e-05 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.235 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10739 3.10 3:0a996d09:::rbd_data.115e380d621c.0000000000000199:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.964810687, 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092681780000000002, 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10739 2024-04-24T00:34:46.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221428+0000", 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0949999999999998e-06 2024-04-24T00:34:46.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221438+0000", 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9310000000000003e-06 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221463+0000", 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5023000000000001e-05 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221477+0000", 2024-04-24T00:34:46.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3591000000000001e-05 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221633+0000", 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156565 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.222026+0000", 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039234399999999998 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.223749+0000", 2024-04-24T00:34:46.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017237109999999999 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230482+0000", 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067323670000000004 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230517+0000", 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981999999999998e-05 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230610+0000", 2024-04-24T00:34:46.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230694+0000", 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4393000000000003e-05 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10768 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:46.241 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.903579783000001, 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092323920000000007, 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10768 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282658+0000", 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1799999999999999e-06 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282664+0000", 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6339999999999998e-06 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282676+0000", 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2320000000000001e-05 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282687+0000", 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1103e-05 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284651+0000", 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00196408 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284827+0000", 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175768 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.285720+0000", 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089352799999999999 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289504+0000", 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003783813 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291726+0000", 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022220339999999999 2024-04-24T00:34:46.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291814+0000", 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7425999999999999e-05 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291889+0000", 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5506000000000005e-05 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10769 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.903495205999999, 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012818598, 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10769 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282742+0000", 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282747+0000", 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0329999999999999e-06 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282759+0000", 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1687e-05 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282768+0000", 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.286885+0000", 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041171139999999998 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.287035+0000", 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149968 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.288104+0000", 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010690540000000001 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295331+0000", 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072273800000000003 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295418+0000", 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295493+0000", 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4825000000000003e-05 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295560+0000", 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7191000000000001e-05 2024-04-24T00:34:46.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10770 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.903455956999998, 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013929258999999999, 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10770 2024-04-24T00:34:46.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282782+0000", 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9000000000000005e-07 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282786+0000", 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1330000000000001e-06 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282795+0000", 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-06 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282803+0000", 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8499999999999994e-06 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289625+0000", 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068213689999999999 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289770+0000", 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014539899999999999 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.290476+0000", 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070591399999999998 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295352+0000", 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048766089999999996 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296546+0000", 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011937580000000001 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296638+0000", 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2179e-05 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296710+0000", 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1527999999999997e-05 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.903422856999999, 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:34:46.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:46.262 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.903386844, 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:46.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:34:46.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:34:46.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:34:46.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.653982450000001, 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.268 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:34:46.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:34:46.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.619945396999999, 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.273 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.0292194329999997, 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.277 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.282 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.0199633080000003, 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30577 3.10 3:089a659d:::rbd_object_map.115e8210e51a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.919578+0000", 2024-04-24T00:34:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.2666583259999999, 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011524019, 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30577 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919578+0000", 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919578+0000", 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919579+0000", 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919584+0000", 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7269999999999998e-06 2024-04-24T00:34:46.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919597+0000", 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2972999999999999e-05 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919606+0000", 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6130000000000004e-06 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.924097+0000", 2024-04-24T00:34:46.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004491381 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.924382+0000", 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028513399999999998 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.925606+0000", 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012234259999999999 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.930883+0000", 2024-04-24T00:34:46.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052770400000000002 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.930936+0000", 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2973000000000003e-05 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931028+0000", 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2650000000000002e-05 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931102+0000", 2024-04-24T00:34:46.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4098999999999996e-05 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30578 3.10 3:089a659d:::rbd_object_map.115e8210e51a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.2666137769999999, 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012659043, 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.292 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30578 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919624+0000", 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919632+0000", 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5969999999999992e-06 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919642+0000", 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8230000000000006e-06 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919650+0000", 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6529999999999994e-06 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.927798+0000", 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0081486619999999992 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.928005+0000", 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020605499999999999 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.929327+0000", 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013227689999999999 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931324+0000", 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019969150000000001 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931986+0000", 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066179100000000001 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.932159+0000", 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017306600000000001 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.932282+0000", 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012284199999999999 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.297 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:46.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:46.300 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:46.296+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1663 write 0x91b35c9 thru 0x91b707c (0x3ab4 bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1665 read 0x7b60cba thru 0x7b66857 (0x5b9e bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1666 writesame 0xb30b61d thru 0xb3121eb data_size 0xd(0x6bcf bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1667 read 0xa68ea7e thru 0xa6930f5 (0x4678 bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1668 write 0xdfc04b1 thru 0xdfd02e3 (0xfe33 bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1670 trunc from 0xdfd02e4 to 0xc4f0ced 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1671 read 0x4ede318 thru 0x4eeb06d (0xcd56 bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1672 punch from 0xaa22f30 to 0xaa22f74, (0x44 bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1673 write 0x8573c34 thru 0x8578c1d (0x4fea bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1675 write 0x9b3a639 thru 0x9b4207f (0x7a47 bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1676 read 0x47c3758 thru 0x47d1abf (0xe368 bytes) 2024-04-24T00:34:46.301 INFO:teuthology.orchestra.run.smithi064.stdout:1677 write 0x3272bab thru 0x327c659 (0x9aaf bytes) 2024-04-24T00:34:46.310 INFO:teuthology.orchestra.run.smithi064.stdout:1678 punch from 0x85f3657 to 0x8600a6e, (0xd417 bytes) 2024-04-24T00:34:46.310 INFO:teuthology.orchestra.run.smithi064.stdout:1680 punch from 0x1b2364c to 0x1b2ba4b, (0x83ff bytes) 2024-04-24T00:34:46.310 INFO:teuthology.orchestra.run.smithi064.stdout:1681 read 0x4d85efc thru 0x4d8df0f (0x8014 bytes) 2024-04-24T00:34:46.310 INFO:teuthology.orchestra.run.smithi064.stdout:1682 punch from 0xb5214f3 to 0xb527ec7, (0x69d4 bytes) 2024-04-24T00:34:46.310 INFO:teuthology.orchestra.run.smithi064.stdout:1683 writesame 0x1ed976a thru 0x1edda54 data_size 0x1(0x42eb bytes) 2024-04-24T00:34:46.358 INFO:teuthology.orchestra.run.smithi064.stdout:1684 writesame 0xe1a4c1f thru 0xe1b0a8a data_size 0xe(0xbe6c bytes) 2024-04-24T00:34:46.358 INFO:teuthology.orchestra.run.smithi064.stdout:1685 write 0xcace532 thru 0xcad2cb0 (0x477f bytes) 2024-04-24T00:34:46.376 INFO:teuthology.orchestra.run.smithi064.stdout:1686 write 0xed10c51 thru 0xed17253 (0x6603 bytes) 2024-04-24T00:34:46.383 INFO:teuthology.orchestra.run.smithi064.stdout:1687 writesame 0x275176b thru 0x275dc1c data_size 0x11(0xc4b2 bytes) 2024-04-24T00:34:46.400 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:46.396+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:46.420 INFO:teuthology.orchestra.run.smithi064.stdout:1689 trunc from 0xed17254 to 0x72bc9aa 2024-04-24T00:34:46.446 INFO:teuthology.orchestra.run.smithi064.stdout:1690 write 0x5e96b9f thru 0x5e9cc21 (0x6083 bytes) 2024-04-24T00:34:46.463 INFO:teuthology.orchestra.run.smithi064.stdout:1691 trunc from 0x72bc9aa to 0x9bc7aa4 2024-04-24T00:34:46.479 INFO:teuthology.orchestra.run.smithi064.stdout:1693 trunc from 0x9bc7aa4 to 0x8fc7a3a 2024-04-24T00:34:46.481 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:46.481 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:46.481 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:46.481 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:46.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:46.516 INFO:teuthology.orchestra.run.smithi064.stdout:1694 writesame 0xbec021e thru 0xbec6eeb data_size 0x13(0x6cce bytes) 2024-04-24T00:34:46.516 INFO:teuthology.orchestra.run.smithi064.stdout:1696 writesame 0x6026b22 thru 0x602d552 data_size 0x1(0x6a31 bytes) 2024-04-24T00:34:46.539 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:46.536+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:46.563 INFO:teuthology.orchestra.run.smithi064.stdout:1697 read 0x62d25bc thru 0x62d4fbc (0x2a01 bytes) 2024-04-24T00:34:46.563 INFO:teuthology.orchestra.run.smithi064.stdout:1698 read 0x6cd4b3c thru 0x6cd7ba3 (0x3068 bytes) 2024-04-24T00:34:46.563 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:46.563 INFO:teuthology.orchestra.run.smithi064.stdout:1701 punch from 0x3999e48 to 0x39a30f8, (0x92b0 bytes) 2024-04-24T00:34:46.563 INFO:teuthology.orchestra.run.smithi064.stdout:1703 punch from 0xb7aabf8 to 0xb7b5608, (0xaa10 bytes) 2024-04-24T00:34:46.563 INFO:teuthology.orchestra.run.smithi064.stdout:1706 writesame 0x54ac015 thru 0x54bbd6c data_size 0x58(0xfd58 bytes) 2024-04-24T00:34:46.569 INFO:teuthology.orchestra.run.smithi064.stdout:1707 read 0x4d7a843 thru 0x4d8162f (0x6ded bytes) 2024-04-24T00:34:46.570 INFO:teuthology.orchestra.run.smithi064.stdout:1708 punch from 0x18d13b6 to 0x18ddffd, (0xcc47 bytes) 2024-04-24T00:34:46.571 INFO:teuthology.orchestra.run.smithi064.stdout:1709 read 0x2da903f thru 0x2da9596 (0x558 bytes) 2024-04-24T00:34:46.571 INFO:teuthology.orchestra.run.smithi064.stdout:1711 write 0xbe615b8 thru 0xbe6f7d1 (0xe21a bytes) 2024-04-24T00:34:46.580 INFO:teuthology.orchestra.run.smithi064.stdout:1713 writesame 0x748bb43 thru 0x7498002 data_size 0x20(0xc4c0 bytes) 2024-04-24T00:34:46.589 INFO:teuthology.orchestra.run.smithi064.stdout:1714 punch from 0x35dd730 to 0x35e8652, (0xaf22 bytes) 2024-04-24T00:34:46.589 INFO:teuthology.orchestra.run.smithi064.stdout:1715 read 0x2c92590 thru 0x2c95110 (0x2b81 bytes) 2024-04-24T00:34:46.590 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:46.590 INFO:teuthology.orchestra.run.smithi064.stdout:1719 writesame 0x2e54260 thru 0x2e5fc1c data_size 0x1(0xb9bd bytes) 2024-04-24T00:34:46.601 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:46.594+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:46.621 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:34:46.671 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:46.671 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:46.671 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:46.671 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:46.671 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:46.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:46.702 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:46.698+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:46.740 INFO:teuthology.orchestra.run.smithi064.stdout:1720 read 0x32a74fb thru 0x32b31d7 (0xbcdd bytes) 2024-04-24T00:34:46.758 INFO:teuthology.orchestra.run.smithi064.stdout:1721 write 0xd3de74c thru 0xd3e9bcd (0xb482 bytes) 2024-04-24T00:34:46.793 INFO:teuthology.orchestra.run.smithi064.stdout:1723 trunc from 0xd3e9bce to 0x234046a 2024-04-24T00:34:46.802 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:46.800+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:46.814 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.885088170000003, 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:46.815 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:34:46.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:34:46.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:34:46.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4984 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.217744222999997, 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024665092, 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4984 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4e-07 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593755+0000", 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6829999999999999e-06 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593765+0000", 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7140000000000005e-06 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593773+0000", 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7829999999999999e-06 2024-04-24T00:34:46.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609097+0000", 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015324669000000001 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609254+0000", 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015632499999999999 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609918+0000", 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066479400000000002 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.615678+0000", 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057599119999999998 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618123+0000", 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024450909999999999 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618280+0000", 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156706 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618415+0000", 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134475 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5029 3.f 3:f41ea3ee:::rbd_data.115ed6c33904.0000000000000006:head [zero 262144~262144,zero 786432~65536,zero 917504~39917,write 957421~8409 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.202292858, 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024904274000000001, 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5029 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609208+0000", 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8959999999999997e-06 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609285+0000", 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6738999999999998e-05 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609308+0000", 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3896999999999998e-05 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609323+0000", 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4543e-05 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622194+0000", 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012870985 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622559+0000", 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036503000000000002 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.623244+0000", 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068464100000000005 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.629548+0000", 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063044329999999999 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.633948+0000", 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044003250000000001 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634037+0000", 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9109000000000004e-05 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634105+0000", 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7675999999999994e-05 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5306 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:46.832 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.126157259000003, 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025510631999999998, 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5306 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685337+0000", 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685342+0000", 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4560000000000002e-06 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685352+0000", 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840000000000006e-06 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685359+0000", 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7100000000000007e-06 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696262+0000", 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010902604 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696476+0000", 2024-04-24T00:34:46.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021439300000000001 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.697460+0000", 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00098361900000000003 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.706972+0000", 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095118609999999999 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710681+0000", 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037093619999999999 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710762+0000", 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0374e-05 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710847+0000", 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5478999999999999e-05 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.837 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10360 3.11 3:8fda792a:::rbd_data.115e380d621c.0000000000000177:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.360103291000001, 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042632046999999999, 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10360 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451393+0000", 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5600000000000001e-06 2024-04-24T00:34:46.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451406+0000", 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2984e-05 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451445+0000", 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9045999999999999e-05 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451461+0000", 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5786000000000001e-05 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451608+0000", 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014694900000000001 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451914+0000", 2024-04-24T00:34:46.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306657 2024-04-24T00:34:46.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:46.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002601744 2024-04-24T00:34:46.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493753+0000", 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039236409 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493841+0000", 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8448000000000004e-05 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493931+0000", 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0198999999999995e-05 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494023+0000", 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1265000000000001e-05 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10360 3.11 e40/21 ondisk, result = 0)", 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.349237998, 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031780810999999999, 2024-04-24T00:34:46.844 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462258+0000", 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2539999999999999e-06 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462266+0000", 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8029999999999997e-06 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462294+0000", 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8637e-05 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462314+0000", 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493814+0000", 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031500133 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493831+0000", 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7357e-05 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494037+0000", 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020520400000000001 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.953122481999998, 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:34:46.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.951873734000003, 2024-04-24T00:34:46.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:34:46.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10590 3.13 e41/21 ondisk, result = 0)", 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:46.858 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.940628150000002, 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042592342999999998, 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870868+0000", 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1730000000000002e-06 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870876+0000", 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2069999999999995e-06 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870900+0000", 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.404e-05 2024-04-24T00:34:46.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870914+0000", 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.366e-05 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913167+0000", 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042252855999999998 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913184+0000", 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7779e-05 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913458+0000", 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027362799999999999 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10592 3.16 e41/21 ondisk, result = 0)", 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.936211554000003, 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044174204000000002, 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.862 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875284+0000", 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1050000000000002e-06 2024-04-24T00:34:46.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875292+0000", 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3200000000000002e-06 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875319+0000", 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7382999999999999e-05 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875334+0000", 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4786e-05 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919179+0000", 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043845151999999998 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919195+0000", 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5846999999999999e-05 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919456+0000", 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026161100000000003 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.794249629999999, 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:34:46.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:34:46.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.780755086999999, 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:34:46.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:34:46.874 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21553 3.7 3:e2dbd82a:::rbd_data.115eeb9272a5.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.715271982000001, 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029891540000000001, 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21553 2024-04-24T00:34:46.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096223+0000", 2024-04-24T00:34:46.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3570000000000001e-06 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096230+0000", 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4080000000000003e-06 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096244+0000", 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4236999999999999e-05 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096251+0000", 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6569999999999997e-06 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096335+0000", 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3997000000000002e-05 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096473+0000", 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013718199999999999 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:46.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.097939+0000", 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014665570000000001 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125952+0000", 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028013198 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125977+0000", 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4613999999999998e-05 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126035+0000", 2024-04-24T00:34:46.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7966e-05 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126113+0000", 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8367000000000004e-05 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:46.880 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.518973972000001, 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:34:46.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:34:46.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21613 3.7 e52/44 ondisk, result = 0)", 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.511140033, 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044307553999999999, 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300355+0000", 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4610000000000001e-06 2024-04-24T00:34:46.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300362+0000", 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4930000000000003e-06 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300382+0000", 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9887000000000001e-05 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300394+0000", 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2799000000000001e-05 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344558+0000", 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044164017999999999 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344589+0000", 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.027e-05 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344661+0000", 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2626000000000006e-05 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:46.891 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:46.892 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.265387829, 2024-04-24T00:34:46.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:34:46.892 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.892 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.892 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.892 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.893 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.893 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.893 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:34:46.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.893 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:46.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:46.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:34:46.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:34:46.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:34:46.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:34:46.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:34:46.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:34:46.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:34:46.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:34:46.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:34:46.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:34:46.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:46.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:34:46.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:34:46.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:34:46.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:34:46.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:46.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:34:46.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:34:46.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:34:46.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:34:46.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:34:46.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:34:46.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:34:46.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:34:46.904 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.904 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.904 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.904 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:46.905 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:46.905 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.237713964999999, 2024-04-24T00:34:46.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:34:46.905 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.905 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.905 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.906 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.906 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:34:46.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.906 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:46.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:46.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:34:46.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:34:46.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.7436911449999997, 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:34:46.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:34:46.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.7346823110000003, 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:34:46.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:34:46.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:34:46.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:34:46.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:34:46.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30153 3.11 3:8dec0a39:::rbd_data.115e8210e51a.0000000000000015:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.6544999230000004, 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026293440000000001, 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30153 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156998+0000", 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7270000000000001e-06 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:46.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157022+0000", 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3958999999999999e-05 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157065+0000", 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3643000000000002e-05 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157090+0000", 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.446e-05 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157200+0000", 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110406 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157303+0000", 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102545 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159693+0000", 2024-04-24T00:34:46.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023907830000000001 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183138+0000", 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023444415999999999 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183198+0000", 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0331999999999998e-05 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:46.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183245+0000", 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7154000000000003e-05 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183287+0000", 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2014999999999997e-05 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:46.928 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:46.930 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:46.924+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:46.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:34:46.932 INFO:teuthology.orchestra.run.smithi064.stdout:1725 read 0x1e6748 thru 0x1ea535 (0x3dee bytes) 2024-04-24T00:34:46.932 INFO:teuthology.orchestra.run.smithi064.stdout:1726 read 0x5f1b92 thru 0x5fd41b (0xb88a bytes) 2024-04-24T00:34:46.932 INFO:teuthology.orchestra.run.smithi064.stdout:1727 write 0x73f23bd thru 0x73f8a29 (0x666d bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1731 read 0x17076e3 thru 0x1710f5d (0x987b bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1732 read 0x6fe5f9b thru 0x6fefbe9 (0x9c4f bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1733 read 0x48e9bc0 thru 0x48f8346 (0xe787 bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1734 punch from 0x5f76deb to 0x5f8484f, (0xda64 bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1735 read 0x324420 thru 0x32af8b (0x6b6c bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1736 punch from 0x54df216 to 0x54ea6bf, (0xb4a9 bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1737 read 0x11238dc thru 0x1128ce4 (0x5409 bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1739 read 0x6717029 thru 0x671aa2b (0x3a03 bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1740 writesame 0xcf3cd50 thru 0xcf46e9c data_size 0x11(0xa14d bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1742 trunc from 0xcf46e9d to 0x9c1a91c 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1743 write 0x19b0fbd thru 0x19b9c97 (0x8cdb bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1744 read 0x1b4b667 thru 0x1b4bf8d (0x927 bytes) 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:46.933 INFO:teuthology.orchestra.run.smithi064.stdout:1747 writesame 0x6fe5db thru 0x706244 data_size 0x46(0x7c6a bytes) 2024-04-24T00:34:46.936 INFO:teuthology.orchestra.run.smithi064.stdout:1748 read 0xb67ab9 thru 0xb76742 (0xec8a bytes) 2024-04-24T00:34:46.937 INFO:teuthology.orchestra.run.smithi064.stdout:1750 read 0x34cca9a thru 0x34dbb7a (0xf0e1 bytes) 2024-04-24T00:34:46.939 INFO:teuthology.orchestra.run.smithi064.stdout:1751 punch from 0x7d76e75 to 0x7d7fef3, (0x907e bytes) 2024-04-24T00:34:46.939 INFO:teuthology.orchestra.run.smithi064.stdout:1752 write 0x4395161 thru 0x439a81c (0x56bc bytes) 2024-04-24T00:34:46.949 INFO:teuthology.orchestra.run.smithi064.stdout:1755 read 0x8edbf9c thru 0x8ee44f6 (0x855b bytes) 2024-04-24T00:34:46.970 INFO:teuthology.orchestra.run.smithi064.stdout:1756 write 0xc152f6f thru 0xc160c8d (0xdd1f bytes) 2024-04-24T00:34:46.979 INFO:teuthology.orchestra.run.smithi064.stdout:1757 write 0x762955e thru 0x7636b3f (0xd5e2 bytes) 2024-04-24T00:34:46.991 INFO:teuthology.orchestra.run.smithi064.stdout:1758 trunc from 0xc160c8e to 0xd42eb38 2024-04-24T00:34:47.015 INFO:teuthology.orchestra.run.smithi064.stdout:1759 trunc from 0xd42eb38 to 0x888d1ff 2024-04-24T00:34:47.031 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:47.028+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:47.041 INFO:teuthology.orchestra.run.smithi064.stdout:1762 punch from 0x8354ab9 to 0x8355ec1, (0x1408 bytes) 2024-04-24T00:34:47.041 INFO:teuthology.orchestra.run.smithi064.stdout:1764 read 0x20cf3cb thru 0x20db2f4 (0xbf2a bytes) 2024-04-24T00:34:47.042 INFO:teuthology.orchestra.run.smithi064.stdout:1765 writesame 0x306b79d thru 0x306d519 data_size 0x1(0x1d7d bytes) 2024-04-24T00:34:47.057 INFO:teuthology.orchestra.run.smithi064.stdout:1767 write 0x148bcba thru 0x1490bf5 (0x4f3c bytes) 2024-04-24T00:34:47.062 INFO:teuthology.orchestra.run.smithi064.stdout:1768 read 0x6107637 thru 0x61141b2 (0xcb7c bytes) 2024-04-24T00:34:47.074 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:47.075 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:47.084 INFO:teuthology.orchestra.run.smithi064.stdout:1769 writesame 0xb07380f thru 0xb0806c1 data_size 0x13(0xceb3 bytes) 2024-04-24T00:34:47.084 INFO:teuthology.orchestra.run.smithi064.stdout:1772 read 0xabb47e1 thru 0xabc24fa (0xdd1a bytes) 2024-04-24T00:34:47.085 INFO:teuthology.orchestra.run.smithi064.stdout:1773 write 0x57935b8 thru 0x57a2e9c (0xf8e5 bytes) 2024-04-24T00:34:47.089 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:34:47.095 INFO:teuthology.orchestra.run.smithi064.stdout:1774 punch from 0x655dfc6 to 0x65685e1, (0xa61b bytes) 2024-04-24T00:34:47.105 INFO:teuthology.orchestra.run.smithi064.stdout:1779 trunc from 0xb0806c2 to 0x4fc10be 2024-04-24T00:34:47.129 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:47.132 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:47.126+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:47.151 INFO:teuthology.orchestra.run.smithi064.stdout:1784 writesame 0x9222e45 thru 0x922d267 data_size 0x1(0xa423 bytes) 2024-04-24T00:34:47.156 INFO:teuthology.orchestra.run.smithi064.stdout:1787 trunc from 0x922d268 to 0x576b8be 2024-04-24T00:34:47.174 INFO:teuthology.orchestra.run.smithi064.stdout:1788 writesame 0x146b603 thru 0x146c41e data_size 0x15(0xe1c bytes) 2024-04-24T00:34:47.177 INFO:teuthology.orchestra.run.smithi064.stdout:1789 write 0x87249b thru 0x87c44e (0x9fb4 bytes) 2024-04-24T00:34:47.196 INFO:teuthology.orchestra.run.smithi064.stdout:1790 trunc from 0x576b8be to 0xae8d74e 2024-04-24T00:34:47.213 INFO:teuthology.orchestra.run.smithi064.stdout:1791 read 0x9747207 thru 0x9747d3e (0xb38 bytes) 2024-04-24T00:34:47.213 INFO:teuthology.orchestra.run.smithi064.stdout:1792 punch from 0x798c469 to 0x798c973, (0x50a bytes) 2024-04-24T00:34:47.213 INFO:teuthology.orchestra.run.smithi064.stdout:1795 writesame 0x4d86f47 thru 0x4d92a60 data_size 0x1b(0xbb1a bytes) 2024-04-24T00:34:47.221 INFO:teuthology.orchestra.run.smithi064.stdout:1797 write 0x4dc84d1 thru 0x4dd84c5 (0xfff5 bytes) 2024-04-24T00:34:47.223 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:47.223 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:47.228 INFO:teuthology.orchestra.run.smithi064.stdout:1798 write 0x8e67ae2 thru 0x8e70bc2 (0x90e1 bytes) 2024-04-24T00:34:47.233 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:47.228+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:47.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:34:47.240 INFO:teuthology.orchestra.run.smithi064.stdout:1799 clone 11 order 20 su 524288 sc 6 2024-04-24T00:34:47.334 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:47.332+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:47.368 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2019 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 1900544~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.833365817000001, 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014241181, 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2019 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530778+0000", 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530779+0000", 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530780+0000", 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.686e-06 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.507e-06 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530790+0000", 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5240000000000002e-06 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544487+0000", 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013696612 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544570+0000", 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2704e-05 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545020+0000", 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044978100000000001 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2020 3.1f 3:fbcd1025:::rbd_data.115eb9c76af6.0000000000000001:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.833347232000001, 2024-04-24T00:34:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014215145, 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2020 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530797+0000", 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9499999999999998e-07 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530799+0000", 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.013e-06 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4289999999999998e-06 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530810+0000", 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4719999999999997e-06 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544812+0000", 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01400174 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544855+0000", 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3217999999999999e-05 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.545012+0000", 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000156878 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10361 3.1f 3:f8601f73:::rbd_data.115e380d621c.0000000000000178:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.911370183000002, 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032855711000000003, 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10361 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452774+0000", 2024-04-24T00:34:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452776+0000", 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.34e-06 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452789+0000", 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2821e-05 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452813+0000", 2024-04-24T00:34:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3903000000000001e-05 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452825+0000", 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1904e-05 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.452918+0000", 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2742000000000001e-05 2024-04-24T00:34:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453066+0000", 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014841100000000001 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00144988 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484060+0000", 2024-04-24T00:34:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029543894000000001 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485505+0000", 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014454610000000001 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485582+0000", 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6792000000000006e-05 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485630+0000", 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7562999999999998e-05 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10362 3.17 3:ea0e3a17:::rbd_data.115e380d621c.0000000000000179:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.910241351000003, 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031846629000000001, 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10362 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453903+0000", 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453904+0000", 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.23e-07 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453909+0000", 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7989999999999999e-06 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453922+0000", 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2911999999999999e-05 2024-04-24T00:34:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.453930+0000", 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1430000000000001e-06 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454001+0000", 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1239e-05 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.454185+0000", 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018420099999999999 2024-04-24T00:34:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.456031+0000", 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018460180000000001 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485548+0000", 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029516936000000001 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485636+0000", 2024-04-24T00:34:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8568999999999999e-05 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485693+0000", 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6307000000000001e-05 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485749+0000", 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6681999999999997e-05 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10361 3.1f e40/21 ondisk, result = 0)", 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.905802823000002, 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025729842999999999, 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458341+0000", 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458343+0000", 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.189e-06 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458348+0000", 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5130000000000004e-06 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458364+0000", 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6235e-05 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.458374+0000", 2024-04-24T00:34:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4220000000000006e-06 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484036+0000", 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025662124000000001 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484049+0000", 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3382e-05 2024-04-24T00:34:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.484071+0000", 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1977999999999999e-05 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10362 3.17 e40/21 ondisk, result = 0)", 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.902725240000002, 2024-04-24T00:34:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024141582000000002, 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461419+0000", 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461421+0000", 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8339999999999999e-06 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461428+0000", 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4939999999999997e-06 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461455+0000", 2024-04-24T00:34:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6571999999999999e-05 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.461468+0000", 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3243e-05 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485528+0000", 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024059515 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485539+0000", 2024-04-24T00:34:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0933e-05 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.485561+0000", 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1991e-05 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:20360 3.1b 3:de2cd4b6:::rbd_data.115ee4742374.0000000000000022:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.867291486999999, 2024-04-24T00:34:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100050336, 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 20360 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496853+0000", 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496855+0000", 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6060000000000001e-06 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496866+0000", 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0308e-05 2024-04-24T00:34:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496891+0000", 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5717000000000001e-05 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.496904+0000", 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2852e-05 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.497012+0000", 2024-04-24T00:34:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107873 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.497126+0000", 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114431 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.503011+0000", 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058843769999999997 2024-04-24T00:34:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596769+0000", 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093757937 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596790+0000", 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1590000000000002e-05 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596851+0000", 2024-04-24T00:34:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1042000000000003e-05 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596903+0000", 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1603e-05 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:20360 3.1b e48/21 ondisk, result = 0)", 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.850177411000001, 2024-04-24T00:34:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082812384000000003, 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513967+0000", 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513968+0000", 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3689999999999999e-06 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513973+0000", 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.673e-06 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.513990+0000", 2024-04-24T00:34:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6806000000000001e-05 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.514003+0000", 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2924e-05 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596727+0000", 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082724848000000004 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596738+0000", 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0501e-05 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.596779+0000", 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1263000000000001e-05 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:20564 3.17 3:ec244480:::rbd_data.115ee4742374.000000000000000d:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.345912946999999, 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081361157000000003, 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 20564 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018231+0000", 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018234+0000", 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.43e-06 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018249+0000", 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4892000000000001e-05 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018285+0000", 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6164999999999997e-05 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018301+0000", 2024-04-24T00:34:47.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5860000000000001e-05 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018472+0000", 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171602 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.018753+0000", 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000280611 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.022778+0000", 2024-04-24T00:34:47.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004025303 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099276+0000", 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076498022999999998 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099401+0000", 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012506299999999999 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099503+0000", 2024-04-24T00:34:47.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101409 2024-04-24T00:34:47.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099592+0000", 2024-04-24T00:34:47.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9798999999999999e-05 2024-04-24T00:34:47.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:20564 3.17 e49/21 ondisk, result = 0)", 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.329249463, 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064713039, 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034895+0000", 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034896+0000", 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.53e-06 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034904+0000", 2024-04-24T00:34:47.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2230000000000007e-06 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034935+0000", 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0278e-05 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.034949+0000", 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.403e-05 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099371+0000", 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064422222000000001 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099390+0000", 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8943000000000001e-05 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.099608+0000", 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217813 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.421 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21501 3.b 3:d2c9feba:::rbd_data.115eeb9272a5.000000000000001b:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.635530695, 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.140289634, 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21501 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728613+0000", 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728616+0000", 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3020000000000002e-06 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728626+0000", 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0045e-05 2024-04-24T00:34:47.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728648+0000", 2024-04-24T00:34:47.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1933000000000002e-05 2024-04-24T00:34:47.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728657+0000", 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6129999999999993e-06 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728726+0000", 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8979000000000005e-05 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.728846+0000", 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012003 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.733780+0000", 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004933242 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868807+0000", 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13502688400000001 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868823+0000", 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6237999999999999e-05 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868863+0000", 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0074999999999998e-05 2024-04-24T00:34:47.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868903+0000", 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0293000000000002e-05 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21501 3.b e52/44 ondisk, result = 0)", 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.614918524, 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11958963, 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749226+0000", 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749227+0000", 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.663e-06 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749235+0000", 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7470000000000005e-06 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749256+0000", 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1243999999999999e-05 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.749268+0000", 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1968e-05 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868685+0000", 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119416835 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868694+0000", 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1139999999999995e-06 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.868815+0000", 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012105899999999999 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21529 3.1b 3:ddb27a62:::rbd_data.115eeb9272a5.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:47.433 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.423191933999998, 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04919722, 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21529 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940952+0000", 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940955+0000", 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7310000000000002e-06 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940971+0000", 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6229999999999999e-05 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.940999+0000", 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7404000000000001e-05 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941012+0000", 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3315e-05 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941093+0000", 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0828999999999995e-05 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.941226+0000", 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133189 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.945656+0000", 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044297529999999998 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989931+0000", 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044275675 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990003+0000", 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2053999999999998e-05 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990084+0000", 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0507999999999996e-05 2024-04-24T00:34:47.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990149+0000", 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5531999999999995e-05 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21529 3.1b e52/21 ondisk, result = 0)", 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.404192289000001, 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030208611, 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.439 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959952+0000", 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959953+0000", 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2720000000000001e-06 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959960+0000", 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4690000000000002e-06 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959979+0000", 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9188999999999999e-05 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.959992+0000", 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2833000000000001e-05 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989981+0000", 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029988982000000001 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.989994+0000", 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3603e-05 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.990160+0000", 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016626299999999999 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.442 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21548 3.6 3:631b2dba:::rbd_data.115eeb9272a5.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.321734514999999, 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035743242000000001, 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21548 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042410+0000", 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042411+0000", 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5700000000000001e-07 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042415+0000", 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5839999999999996e-06 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042427+0000", 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1504e-05 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042432+0000", 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4319999999999998e-06 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042481+0000", 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8557999999999998e-05 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.042573+0000", 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2349999999999995e-05 2024-04-24T00:34:47.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.045702+0000", 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003128858 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.077952+0000", 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032250612999999997 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078015+0000", 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2855999999999996e-05 2024-04-24T00:34:47.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078092+0000", 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6922000000000007e-05 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078153+0000", 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0708000000000002e-05 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21548 3.6 e52/21 ondisk, result = 0)", 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.307987235999999, 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022006369000000001, 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056157+0000", 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056158+0000", 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056163+0000", 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8300000000000003e-06 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056180+0000", 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6575999999999999e-05 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.056188+0000", 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6219999999999998e-06 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.077995+0000", 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021806968999999999 2024-04-24T00:34:47.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078007+0000", 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1771e-05 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.078163+0000", 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015619900000000001 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22146 3.18 3:1e672920:::rbd_data.115e6a380959.0000000000000004:head [zero 0~262144,write 262144~262144 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:47.452 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.557288141000001, 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026558528000000001, 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22146 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806856+0000", 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.806857+0000", 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.384e-06 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807101+0000", 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024388599999999999 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807118+0000", 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6364000000000001e-05 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807127+0000", 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9469999999999992e-06 2024-04-24T00:34:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807208+0000", 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0982999999999994e-05 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807431+0000", 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022303899999999999 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.808060+0000", 2024-04-24T00:34:47.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062940900000000004 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832854+0000", 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024794415 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833269+0000", 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041483500000000001 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833342+0000", 2024-04-24T00:34:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2755000000000004e-05 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833415+0000", 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2510999999999997e-05 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22147 3.18 3:1b6fdb14:::rbd_data.115e6a380959.0000000000000002:head [write 0~482922 [fadvise_sequential+fadvise_nocache],zero 482922~97123,write 580045~47154 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.556989227000001, 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028316734999999999, 2024-04-24T00:34:47.458 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22147 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807155+0000", 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807156+0000", 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6499999999999998e-07 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807545+0000", 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038943700000000002 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807561+0000", 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5724999999999999e-05 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.807570+0000", 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1679999999999993e-06 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.820773+0000", 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013202667 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.820953+0000", 2024-04-24T00:34:47.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018013799999999999 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.821279+0000", 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000326244 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.833920+0000", 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012641316999999999 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835320+0000", 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013997219999999999 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835412+0000", 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2283999999999999e-05 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.835472+0000", 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9268000000000002e-05 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:22148 3.b 3:d51ed8a7:::rbd_data.115e6a380959.0000000000000005:head [write 0~524288 [fadvise_sequential+fadvise_nocache],zero 1048576~524288,zero 5242880~89944,write 5332824~63306 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.554393603999999, 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01907449, 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 22148 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809751+0000", 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.809752+0000", 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.764e-06 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810284+0000", 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053170400000000003 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810302+0000", 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7723e-05 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810313+0000", 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1486e-05 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810413+0000", 2024-04-24T00:34:47.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9906000000000002e-05 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810646+0000", 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023295499999999999 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.810942+0000", 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000296218 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.823674+0000", 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012731732000000001 2024-04-24T00:34:47.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828700+0000", 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050263729999999998 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828771+0000", 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1027999999999998e-05 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.828825+0000", 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600999999999997e-05 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.468 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:22146 3.18 e58/44 ondisk, result = 0)", 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.550599630000001, 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01931886, 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813545+0000", 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813546+0000", 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813553+0000", 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1369999999999996e-06 2024-04-24T00:34:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813574+0000", 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1185000000000001e-05 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.813584+0000", 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0297e-05 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832839+0000", 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019254335000000001 2024-04-24T00:34:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832848+0000", 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9609999999999994e-06 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:32.832863+0000", 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5722e-05 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.472 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:47.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:47.474 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:47.470+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:47.575 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:47.572+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:47.652 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:47.652 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:47.652 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:47.652 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:47.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:47.714 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:47.710+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:47.776 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:47.770+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:47.798 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:47.798 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:47.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:47.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:47.897 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:47.894+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:47.978 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:47.979 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:47.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:47.979 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:47.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.979 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2047 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:47.979 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:47.979 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.444099143000003, 2024-04-24T00:34:47.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010295987, 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2047 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530472+0000", 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2799999999999995e-07 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530475+0000", 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0939999999999999e-06 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530485+0000", 2024-04-24T00:34:47.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7969999999999995e-06 2024-04-24T00:34:47.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530491+0000", 2024-04-24T00:34:47.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8709999999999999e-06 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540514+0000", 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010022499000000001 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540626+0000", 2024-04-24T00:34:47.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111981 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540767+0000", 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014091700000000001 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2048 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.444068332000001, 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010591068, 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2048 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2699999999999999e-07 2024-04-24T00:34:47.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530520+0000", 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8165999999999999e-05 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530533+0000", 2024-04-24T00:34:47.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2065999999999999e-05 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540812+0000", 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010266239 2024-04-24T00:34:47.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540925+0000", 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112676 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541093+0000", 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016786500000000001 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2049 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.444012088999997, 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010815047, 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2049 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530562+0000", 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4369999999999999e-06 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530572+0000", 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530579+0000", 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:34:47.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541138+0000", 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010558209000000001 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541239+0000", 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101483 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541373+0000", 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133769 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2050 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443980740000001, 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011052364, 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:47.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2050 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:47.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530590+0000", 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6599999999999996e-07 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:47.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530593+0000", 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.937e-06 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530602+0000", 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4360000000000008e-06 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530609+0000", 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0090000000000001e-06 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:47.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541411+0000", 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01080191 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541511+0000", 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9839000000000004e-05 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541642+0000", 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130667 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2051 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443951380000001, 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011318002000000001, 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2051 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.001 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2900000000000004e-07 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530622+0000", 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9490000000000001e-06 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530632+0000", 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999994e-06 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.131e-06 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541683+0000", 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011044629 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120383 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541937+0000", 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132941 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2052 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443922180000001, 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011583872, 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2052 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.44e-07 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530658+0000", 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9850000000000001e-06 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530668+0000", 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530675+0000", 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6240000000000002e-06 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541978+0000", 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011302921 2024-04-24T00:34:48.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542100+0000", 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121593 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542232+0000", 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013176600000000001 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2077 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443884779999998, 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011842221, 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2077 2024-04-24T00:34:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:48.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6000000000000003e-07 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.095e-06 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530699+0000", 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530706+0000", 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2429999999999999e-06 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542275+0000", 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011568746 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542386+0000", 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011176200000000001 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542527+0000", 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014114499999999999 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2078 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443854453999997, 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012081010999999999, 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2078 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2200000000000004e-07 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530719+0000", 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9349999999999999e-06 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530728+0000", 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0860000000000008e-06 2024-04-24T00:34:48.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530735+0000", 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2169999999999997e-06 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542568+0000", 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011832540000000001 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542667+0000", 2024-04-24T00:34:48.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9000999999999996e-05 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542797+0000", 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012971 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2079 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443825140999998, 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012334128999999999, 2024-04-24T00:34:48.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2079 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530748+0000", 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9040000000000001e-06 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2159999999999995e-06 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530764+0000", 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9890000000000003e-06 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542839+0000", 2024-04-24T00:34:48.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012074354000000001 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542945+0000", 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106443 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543079+0000", 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013369599999999999 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2080 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443796341000002, 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012588825999999999, 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2080 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.024 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530777+0000", 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.022e-06 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:48.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9879999999999993e-06 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530793+0000", 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1409999999999999e-06 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543122+0000", 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012328274 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543230+0000", 2024-04-24T00:34:48.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108046 2024-04-24T00:34:48.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543362+0000", 2024-04-24T00:34:48.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013286799999999999 2024-04-24T00:34:48.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2081 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443767456000003, 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012843515, 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2081 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:48.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1500000000000005e-07 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.819e-06 2024-04-24T00:34:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530815+0000", 2024-04-24T00:34:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3540000000000006e-06 2024-04-24T00:34:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530835+0000", 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9983000000000002e-05 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543405+0000", 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012569739 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543513+0000", 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010821500000000001 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543646+0000", 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289000000000001 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2082 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443724240999998, 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013081859, 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2082 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530849+0000", 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0750000000000002e-06 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530859+0000", 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7079999999999992e-06 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530866+0000", 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4309999999999997e-06 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543689+0000", 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012822750000000001 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543796+0000", 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106526 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543928+0000", 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131846 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2083 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443693967999998, 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013453964000000001, 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2083 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530877+0000", 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530887+0000", 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186e-05 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530896+0000", 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5480000000000007e-06 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530904+0000", 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4599999999999997e-06 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543970+0000", 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013066067000000001 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544178+0000", 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020785000000000001 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544330+0000", 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015230699999999999 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2084 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.443656144000002, 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907047000000001, 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2084 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3099999999999998e-07 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530922+0000", 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.498e-06 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530932+0000", 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530940+0000", 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4769999999999997e-06 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544421+0000", 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013481739 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544673+0000", 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025143600000000001 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544821+0000", 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014819 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.540578779000001, 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:34:48.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:34:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.485067293, 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:48.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33687 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.5915875989999999, 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015720043, 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33687 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382984+0000", 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382994+0000", 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.008e-05 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383020+0000", 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5565999999999999e-05 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383034+0000", 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383167+0000", 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132719 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383325+0000", 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015864899999999999 2024-04-24T00:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.384182+0000", 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085650199999999998 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398477+0000", 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014294836 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398509+0000", 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2731999999999998e-05 2024-04-24T00:34:48.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398615+0000", 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106084 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398702+0000", 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7068000000000001e-05 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33687 3.6 e70/59 ondisk, result = 0)", 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.587126663, 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011051082, 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387445+0000", 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387455+0000", 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387483+0000", 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8181999999999999e-05 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387497+0000", 2024-04-24T00:34:48.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.401e-05 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398435+0000", 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010938296 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398464+0000", 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8881999999999999e-05 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398494+0000", 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0219e-05 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.3844906450000001, 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:34:48.070 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:34:48.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:34:48.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.342925691, 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:48.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:34:48.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:48.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:48.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:48.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:34:48.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:34:48.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:48.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:48.080 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:48.081 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:47.976+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:48.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:48.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:48.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:48.083 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:48.083 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:48.083 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:48.084 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:48.084 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:48.084 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:48.084 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:48.085 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:48.085 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:48.078+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:48.101 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:48.101 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:48.123 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:48.124 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:48.146 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:48.146 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:48.183 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:48.180+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:48.221 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:48.222 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:48.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:48.285 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:48.282+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:48.362 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:48.362 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:48.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:48.418 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:48.414+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:48.486 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:48.482+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:48.545 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:48.545 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:48.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:48.587 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:48.584+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:48.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:34:48.689 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:48.684+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:48.750 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:48.750 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:48.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:48.789 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:48.784+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:48.890 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:48.886+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:48.912 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:48.912 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:48.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:49.013 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:49.010+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:49.091 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:49.092 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:49.092 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:49.088+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:49.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:49.193 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:49.188+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:49.236 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:49.237 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:49.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:49.295 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:49.290+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:49.395 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:49.390+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:49.415 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:49.415 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:49.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:49.495 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:49.490+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:49.539 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:34:49.555 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:49.555 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:49.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:49.615 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:49.612+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:49.688 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:49.688 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:49.697 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:49.690+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:49.703 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:49.723 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-04-24T00:34:49.798 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:49.792+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:49.873 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:49.873 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:49.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:49.931 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:49.928+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:50.000 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:34:49.994+0000 7f0258295640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25166) UID: 0 2024-04-24T00:34:50.054 INFO:teuthology.orchestra.run.smithi064.stdout:1800 write 0xa7f6d38 thru 0xa7fe27a (0x7543 bytes) 2024-04-24T00:34:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:50.069 INFO:teuthology.orchestra.run.smithi064.stdout:1805 read 0x9feb322 thru 0x9ff5b2a (0xa809 bytes) 2024-04-24T00:34:50.069 INFO:teuthology.orchestra.run.smithi064.stdout:1806 write 0x512c711 thru 0x5132d13 (0x6603 bytes) 2024-04-24T00:34:50.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:50.077 INFO:teuthology.orchestra.run.smithi064.stdout:1807 writesame 0x918fcbd thru 0x919ce96 data_size 0x1(0xd1da bytes) 2024-04-24T00:34:50.101 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:50.094+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:50.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:50.198 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:50.199 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:50.206 INFO:teuthology.orchestra.run.smithi064.stdout:1808 read 0x400888 thru 0x40c247 (0xb9c0 bytes) 2024-04-24T00:34:50.206 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:50.202+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:50.207 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:50.207 INFO:teuthology.orchestra.run.smithi064.stdout:1811 punch from 0x975fa18 to 0x976d8e1, (0xdec9 bytes) 2024-04-24T00:34:50.207 INFO:teuthology.orchestra.run.smithi064.stdout:1812 write 0x1f51d32 thru 0x1f59d68 (0x8037 bytes) 2024-04-24T00:34:50.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:50.215 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:50.215 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:50.225 INFO:teuthology.orchestra.run.smithi064.stdout:1815 write 0xd99583a thru 0xd998109 (0x28d0 bytes) 2024-04-24T00:34:50.230 INFO:teuthology.orchestra.run.smithi064.stdout:1816 read 0x5fe7fc2 thru 0x5ff3d04 (0xbd43 bytes) 2024-04-24T00:34:50.230 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:50.230 INFO:teuthology.orchestra.run.smithi064.stdout:1818 write 0x3abda0e thru 0x3acbfa8 (0xe59b bytes) 2024-04-24T00:34:50.241 INFO:teuthology.orchestra.run.smithi064.stdout:1819 read 0xc16421d thru 0xc16ab9c (0x6980 bytes) 2024-04-24T00:34:50.243 INFO:teuthology.orchestra.run.smithi064.stdout:1822 trunc from 0xd99810a to 0xed33c21 2024-04-24T00:34:50.250 INFO:teuthology.orchestra.run.smithi064.stdout:1823 writesame 0xdc59f07 thru 0xdc5a2b7 data_size 0x1b(0x3b1 bytes) 2024-04-24T00:34:50.256 INFO:teuthology.orchestra.run.smithi064.stdout:1825 writesame 0x88107be thru 0x881cf97 data_size 0x3(0xc7da bytes) 2024-04-24T00:34:50.303 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:50.298+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:50.310 INFO:teuthology.orchestra.run.smithi064.stdout:1827 trunc from 0xed33c21 to 0x92defe3 2024-04-24T00:34:50.318 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:34:50.318 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:34:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:1829 write 0x5245e5e thru 0x5252795 (0xc938 bytes) 2024-04-24T00:34:50.335 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:50.335 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:50.338 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:50.338 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:50.347 INFO:teuthology.orchestra.run.smithi064.stdout:1830 write 0xb7aab3a thru 0xb7b04d1 (0x5998 bytes) 2024-04-24T00:34:50.351 INFO:teuthology.orchestra.run.smithi064.stdout:1831 punch from 0x6eea3 to 0x7ec34, (0xfd91 bytes) 2024-04-24T00:34:50.351 INFO:teuthology.orchestra.run.smithi064.stdout:1833 write 0x4ede6f0 thru 0x4eea3f4 (0xbd05 bytes) 2024-04-24T00:34:50.353 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:50.354 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:50.354 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:50.359 INFO:teuthology.orchestra.run.smithi064.stdout:1834 punch from 0x70cde1e to 0x70d1f79, (0x415b bytes) 2024-04-24T00:34:50.359 INFO:teuthology.orchestra.run.smithi064.stdout:1835 read 0x6ece751 thru 0x6ed5212 (0x6ac2 bytes) 2024-04-24T00:34:50.359 INFO:teuthology.orchestra.run.smithi064.stdout:1839 read 0x1e5c3d7 thru 0x1e63641 (0x726b bytes) 2024-04-24T00:34:50.369 INFO:teuthology.orchestra.run.smithi064.stdout:1841 trunc from 0xb7b04d2 to 0x55470c6 2024-04-24T00:34:50.371 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:50.371 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:50.396 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:50.404 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:50.400+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:50.405 INFO:teuthology.orchestra.run.smithi064.stdout:1842 punch from 0xe2aa20 to 0xe2c74e, (0x1d2e bytes) 2024-04-24T00:34:50.405 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:50.405 INFO:teuthology.orchestra.run.smithi064.stdout:1845 read 0x3f4b4b8 thru 0x3f5abeb (0xf734 bytes) 2024-04-24T00:34:50.405 INFO:teuthology.orchestra.run.smithi064.stdout:1846 punch from 0x3769d08 to 0x376cc04, (0x2efc bytes) 2024-04-24T00:34:50.414 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:50.415 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:50.424 INFO:teuthology.orchestra.run.smithi064.stdout:1847 write 0xc216d7a thru 0xc21eb4f (0x7dd6 bytes) 2024-04-24T00:34:50.429 INFO:teuthology.orchestra.run.smithi064.stdout:1849 read 0x2fb65fe thru 0x2fc1943 (0xb346 bytes) 2024-04-24T00:34:50.429 INFO:teuthology.orchestra.run.smithi064.stdout:1850 write 0x22154cb thru 0x2221515 (0xc04b bytes) 2024-04-24T00:34:50.434 INFO:teuthology.orchestra.run.smithi064.stdout:1851 write 0x798c71f thru 0x798caf1 (0x3d3 bytes) 2024-04-24T00:34:50.438 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:50.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:50.438 INFO:teuthology.orchestra.run.smithi064.stdout:1852 read 0xb4733d0 thru 0xb47658e (0x31bf bytes) 2024-04-24T00:34:50.438 INFO:teuthology.orchestra.run.smithi064.stdout:1854 write 0x91db68a thru 0x91e81a1 (0xcb18 bytes) 2024-04-24T00:34:50.445 INFO:teuthology.orchestra.run.smithi064.stdout:1855 read 0x11b94a8 thru 0x11bc20f (0x2d68 bytes) 2024-04-24T00:34:50.448 INFO:teuthology.orchestra.run.smithi064.stdout:1856 trunc from 0xc21eb50 to 0xa5caf72 2024-04-24T00:34:50.459 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:50.459 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:50.461 INFO:teuthology.orchestra.run.smithi064.stdout:1858 read 0x225ef19 thru 0x226eeee (0xffd6 bytes) 2024-04-24T00:34:50.461 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:50.461 INFO:teuthology.orchestra.run.smithi064.stdout:1863 punch from 0xe5209e to 0xe57d5e, (0x5cc0 bytes) 2024-04-24T00:34:50.462 INFO:teuthology.orchestra.run.smithi064.stdout:1864 write 0x36737bc thru 0x3681e51 (0xe696 bytes) 2024-04-24T00:34:50.472 INFO:teuthology.orchestra.run.smithi064.stdout:1865 punch from 0x228087a to 0x2284528, (0x3cae bytes) 2024-04-24T00:34:50.472 INFO:teuthology.orchestra.run.smithi064.stdout:1866 read 0x2c33f85 thru 0x2c3c527 (0x85a3 bytes) 2024-04-24T00:34:50.473 INFO:teuthology.orchestra.run.smithi064.stdout:1867 punch from 0x83534ce to 0x835cd27, (0x9859 bytes) 2024-04-24T00:34:50.473 INFO:teuthology.orchestra.run.smithi064.stdout:1868 write 0x1c04751 thru 0x1c095e2 (0x4e92 bytes) 2024-04-24T00:34:50.490 INFO:teuthology.orchestra.run.smithi064.stdout:1870 write 0xb99c73d thru 0xb99d22b (0xaef bytes) 2024-04-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.491 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:5336 3.1d e36/21 3:b83bad9e:::rbd_data.115ed6c33904.0000000000000086:head v 36'3, mlcod=34'1)", 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.790285587, 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010615080000000001, 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694758+0000", 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694760+0000", 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8080000000000001e-06 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694891+0000", 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013035600000000001 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694916+0000", 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5392e-05 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.694930+0000", 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4059999999999999e-05 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695055+0000", 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124499 2024-04-24T00:34:50.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.695112+0000", 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7853000000000003e-05 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705273+0000", 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010160626000000001 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.705373+0000", 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100486 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10257 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.282641628, 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011690176, 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.496 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10257 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202402+0000", 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202404+0000", 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.897e-06 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202412+0000", 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5499999999999997e-06 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202430+0000", 2024-04-24T00:34:50.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8043e-05 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.202442+0000", 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439999999999999e-05 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207283+0000", 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048410600000000003 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.207552+0000", 2024-04-24T00:34:50.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026864100000000002 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.208755+0000", 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012028189999999999 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.211687+0000", 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029325240000000002 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.213944+0000", 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022567300000000002 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214023+0000", 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8932999999999998e-05 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214093+0000", 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9538999999999999e-05 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10288 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.152298213999998, 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.009564744, 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10288 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332746+0000", 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332747+0000", 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:50.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332769+0000", 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1903e-05 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332780+0000", 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1243000000000001e-05 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.332789+0000", 2024-04-24T00:34:50.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0699999999999996e-06 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336540+0000", 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037510339999999999 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.336760+0000", 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021991099999999999 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.337418+0000", 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000658127 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.339101+0000", 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016826390000000001 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342150+0000", 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030494850000000002 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342236+0000", 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6212999999999996e-05 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.342311+0000", 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4228999999999997e-05 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.507 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10365 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.990271995000001, 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098439340000000004, 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10365 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494772+0000", 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494773+0000", 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1399999999999995e-07 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494782+0000", 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3649999999999993e-06 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494794+0000", 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1606e-05 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494804+0000", 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0013999999999999e-05 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498700+0000", 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038961310000000002 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.498969+0000", 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000269432 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.499771+0000", 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080119200000000005 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.503502+0000", 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037318300000000002 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504304+0000", 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080115000000000004 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504475+0000", 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017146199999999999 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.504616+0000", 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014083800000000001 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10409 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.908689793999997, 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010506112, 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.513 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10409 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576354+0000", 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576355+0000", 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576359+0000", 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3229999999999999e-06 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576372+0000", 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.576383+0000", 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1192999999999999e-05 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580539+0000", 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041557230000000001 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.580813+0000", 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027430400000000001 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.582037+0000", 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001223879 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.584201+0000", 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002164408 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586567+0000", 2024-04-24T00:34:50.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023657090000000001 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586731+0000", 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016439800000000001 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.586860+0000", 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128835 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10645 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.457685712, 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010437706999999999, 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10645 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027358+0000", 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027359+0000", 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.06e-06 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027365+0000", 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2769999999999996e-06 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027380+0000", 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5537e-05 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.027389+0000", 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5299999999999996e-06 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.030838+0000", 2024-04-24T00:34:50.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003449232 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031115+0000", 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000277254 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.031925+0000", 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00080932599999999997 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.035618+0000", 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003693647 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037627+0000", 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020091390000000001 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037720+0000", 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2775000000000005e-05 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.037796+0000", 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.593e-05 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10719 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.306514544999999, 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010945605000000001, 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.524 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10719 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178529+0000", 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178530+0000", 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1800000000000005e-07 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178535+0000", 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5129999999999998e-06 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178544+0000", 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2359999999999993e-06 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.178550+0000", 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.421e-06 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181685+0000", 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031346989999999999 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.181965+0000", 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027960399999999997 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.182619+0000", 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00065424799999999996 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.186425+0000", 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038062690000000001 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189312+0000", 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028871809999999999 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189407+0000", 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4345000000000006e-05 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.189475+0000", 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8270999999999997e-05 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10739 3.10 3:0a996d09:::rbd_data.115e380d621c.0000000000000199:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.263618043000001, 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092681780000000002, 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10739 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.530 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221426+0000", 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221428+0000", 2024-04-24T00:34:50.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0949999999999998e-06 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221438+0000", 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9310000000000003e-06 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221463+0000", 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5023000000000001e-05 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221477+0000", 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3591000000000001e-05 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.221633+0000", 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156565 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.222026+0000", 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00039234399999999998 2024-04-24T00:34:50.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.223749+0000", 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017237109999999999 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230482+0000", 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067323670000000004 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230517+0000", 2024-04-24T00:34:50.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4981999999999998e-05 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230610+0000", 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.230694+0000", 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4393000000000003e-05 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10768 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.202387139000002, 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0092323920000000007, 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10768 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.536 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282657+0000", 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282658+0000", 2024-04-24T00:34:50.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1799999999999999e-06 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282664+0000", 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6339999999999998e-06 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282676+0000", 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2320000000000001e-05 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282687+0000", 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1103e-05 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284651+0000", 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00196408 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.284827+0000", 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175768 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.285720+0000", 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089352799999999999 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289504+0000", 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003783813 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291726+0000", 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022220339999999999 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291814+0000", 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7425999999999999e-05 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291889+0000", 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5506000000000005e-05 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10769 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:50.541 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.202302562, 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012818598, 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10769 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282742+0000", 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282747+0000", 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0329999999999999e-06 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282759+0000", 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1687e-05 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282768+0000", 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.286885+0000", 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041171139999999998 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.287035+0000", 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149968 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.288104+0000", 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010690540000000001 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295331+0000", 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072273800000000003 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295418+0000", 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295493+0000", 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4825000000000003e-05 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295560+0000", 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7191000000000001e-05 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10770 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.202263313000003, 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013929258999999999, 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.547 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10770 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282782+0000", 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9000000000000005e-07 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282786+0000", 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1330000000000001e-06 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282795+0000", 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-06 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282803+0000", 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8499999999999994e-06 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289625+0000", 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068213689999999999 2024-04-24T00:34:50.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289770+0000", 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014539899999999999 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.290476+0000", 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070591399999999998 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295352+0000", 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048766089999999996 2024-04-24T00:34:50.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296546+0000", 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011937580000000001 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296638+0000", 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2179e-05 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296710+0000", 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1527999999999997e-05 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.552 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.202230213, 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.202194200000001, 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.558 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:34:50.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:34:50.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.952789805999998, 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:34:50.563 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:34:50.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:34:50.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:34:50.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.918752753, 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:34:50.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:34:50.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.3280267890000008, 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.572 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:34:50.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:34:50.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:34:50.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.3187706640000005, 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:34:50.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30577 3.10 3:089a659d:::rbd_object_map.115e8210e51a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.919578+0000", 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.5654656819999992, 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011524019, 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.582 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30577 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919578+0000", 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919578+0000", 2024-04-24T00:34:50.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919579+0000", 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919584+0000", 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7269999999999998e-06 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919597+0000", 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2972999999999999e-05 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919606+0000", 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6130000000000004e-06 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.924097+0000", 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004491381 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.924382+0000", 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028513399999999998 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.925606+0000", 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012234259999999999 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.930883+0000", 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052770400000000002 2024-04-24T00:34:50.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.930936+0000", 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2973000000000003e-05 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931028+0000", 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2650000000000002e-05 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931102+0000", 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4098999999999996e-05 2024-04-24T00:34:50.587 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30578 3.10 3:089a659d:::rbd_object_map.115e8210e51a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.5654211329999992, 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012659043, 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.588 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30578 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919624+0000", 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919632+0000", 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5969999999999992e-06 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919642+0000", 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8230000000000006e-06 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919650+0000", 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6529999999999994e-06 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.927798+0000", 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0081486619999999992 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.928005+0000", 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020605499999999999 2024-04-24T00:34:50.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.929327+0000", 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013227689999999999 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931324+0000", 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019969150000000001 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931986+0000", 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066179100000000001 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.932159+0000", 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017306600000000001 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.932282+0000", 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012284199999999999 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33609 3.a 3:5011f56c:::rbd_data.115ec4f9b3a7.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:50.593 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:46.164819+0000", 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.3202252019999996, 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011459329000000001, 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33609 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:50.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.164819+0000", 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.164819+0000", 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.164820+0000", 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.116e-06 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:50.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.164825+0000", 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5860000000000004e-06 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.164840+0000", 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4034e-05 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.164848+0000", 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4749999999999993e-06 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.164902+0000", 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3726e-05 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.164961+0000", 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8987999999999998e-05 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.165576+0000", 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00061543600000000002 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.176137+0000", 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01056115 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.176158+0000", 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0411000000000002e-05 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.176217+0000", 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9713999999999998e-05 2024-04-24T00:34:50.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.176278+0000", 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0693000000000003e-05 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:50.599 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:50.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:50.602 INFO:teuthology.orchestra.run.smithi064.stdout:1872 punch from 0xc508ed to 0xc55012, (0x4725 bytes) 2024-04-24T00:34:50.602 INFO:teuthology.orchestra.run.smithi064.stdout:1873 trunc from 0xb99d22c to 0x9828a39 2024-04-24T00:34:50.602 INFO:teuthology.orchestra.run.smithi064.stdout:1874 writesame 0x2877c57 thru 0x287cf42 data_size 0x1d(0x52ec bytes) 2024-04-24T00:34:50.602 INFO:teuthology.orchestra.run.smithi064.stdout:1876 punch from 0x132b75a to 0x132d0ff, (0x19a5 bytes) 2024-04-24T00:34:50.602 INFO:teuthology.orchestra.run.smithi064.stdout:1878 writesame 0x3e11270 thru 0x3e1a833 data_size 0x1b(0x95c4 bytes) 2024-04-24T00:34:50.602 INFO:teuthology.orchestra.run.smithi064.stdout:1879 writesame 0xb894e78 thru 0xb89f982 data_size 0x1(0xab0b bytes) 2024-04-24T00:34:50.603 INFO:teuthology.orchestra.run.smithi064.stdout:1880 trunc from 0xb89f983 to 0x7053cb7 2024-04-24T00:34:50.603 INFO:teuthology.orchestra.run.smithi064.stdout:1881 read 0x605bee thru 0x608292 (0x26a5 bytes) 2024-04-24T00:34:50.603 INFO:teuthology.orchestra.run.smithi064.stdout:1882 write 0x9a93036 thru 0x9aa107d (0xe048 bytes) 2024-04-24T00:34:50.603 INFO:teuthology.orchestra.run.smithi064.stdout:1883 write 0x89b4427 thru 0x89be6c3 (0xa29d bytes) 2024-04-24T00:34:50.603 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:50.594+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:50.612 INFO:teuthology.orchestra.run.smithi064.stdout:1885 writesame 0xc35ef2f thru 0xc3696cc data_size 0x23(0xa79e bytes) 2024-04-24T00:34:50.612 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:50.613 INFO:teuthology.orchestra.run.smithi064.stdout:1890 write 0x9258122 thru 0x9259be9 (0x1ac8 bytes) 2024-04-24T00:34:50.631 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:50.631 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:50.631 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:50.631 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-24T00:34:50.632 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:34:50.634 INFO:teuthology.orchestra.run.smithi064.stdout:1892 write 0xd3ebb76 thru 0xd3f848a (0xc915 bytes) 2024-04-24T00:34:50.640 INFO:teuthology.orchestra.run.smithi064.stdout:1894 write 0xb3558df thru 0xb35b2d3 (0x59f5 bytes) 2024-04-24T00:34:50.648 INFO:teuthology.orchestra.run.smithi064.stdout:1895 read 0x82f27ac thru 0x82faac9 (0x831e bytes) 2024-04-24T00:34:50.657 INFO:teuthology.orchestra.run.smithi064.stdout:1897 trunc from 0xd3f848b to 0x9202ffb 2024-04-24T00:34:50.675 INFO:teuthology.orchestra.run.smithi064.stdout:1898 write 0x190d4bd thru 0x191c3c4 (0xef08 bytes) 2024-04-24T00:34:50.680 INFO:teuthology.orchestra.run.smithi064.stdout:1899 read 0x37d27df thru 0x37e128a (0xeaac bytes) 2024-04-24T00:34:50.681 INFO:teuthology.orchestra.run.smithi064.stdout:1901 punch from 0x6edfe51 to 0x6ee906c, (0x921b bytes) 2024-04-24T00:34:50.682 INFO:teuthology.orchestra.run.smithi064.stdout:1904 writesame 0x57bb558 thru 0x57c95d4 data_size 0x1(0xe07d bytes) 2024-04-24T00:34:50.698 INFO:tasks.ceph.osd.0:Stopped 2024-04-24T00:34:50.698 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-04-24T00:34:50.698 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:34:50.698 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-04-24T00:34:50.701 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:50.696+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:50.742 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T00:34:50.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-04-24T00:34:50.780 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:50.780 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:50.780 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:50.780 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:50.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:50.829 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:50.826+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:50.835 INFO:teuthology.orchestra.run.smithi064.stdout:1905 writesame 0x999038e thru 0x9994e3f data_size 0x3(0x4ab2 bytes) 2024-04-24T00:34:50.835 INFO:teuthology.orchestra.run.smithi064.stdout:1906 punch from 0x13b1ebb to 0x13b5774, (0x38b9 bytes) 2024-04-24T00:34:50.835 INFO:teuthology.orchestra.run.smithi064.stdout:1907 read 0x1dcae62 thru 0x1dd210b (0x72aa bytes) 2024-04-24T00:34:50.835 INFO:teuthology.orchestra.run.smithi064.stdout:1908 writesame 0x4d77785 thru 0x4d84ffe data_size 0x16(0xd87a bytes) 2024-04-24T00:34:50.840 INFO:teuthology.orchestra.run.smithi064.stdout:1910 read 0x208f6ed thru 0x209daaa (0xe3be bytes) 2024-04-24T00:34:50.841 INFO:teuthology.orchestra.run.smithi064.stdout:1912 write 0x1c45bc thru 0x1c7dc7 (0x380c bytes) 2024-04-24T00:34:50.843 INFO:teuthology.orchestra.run.smithi064.stdout:1913 writesame 0x4498a19 thru 0x44a88e5 data_size 0x11(0xfecd bytes) 2024-04-24T00:34:50.850 INFO:teuthology.orchestra.run.smithi064.stdout:1914 read 0x6c6e437 thru 0x6c6f73a (0x1304 bytes) 2024-04-24T00:34:50.862 INFO:teuthology.orchestra.run.smithi064.stdout:1915 trunc from 0x9994e40 to 0x34447fb 2024-04-24T00:34:50.883 INFO:teuthology.orchestra.run.smithi064.stdout:1917 read 0x110a5f3 thru 0x1116cf2 (0xc700 bytes) 2024-04-24T00:34:50.903 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:50.900+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:50.909 INFO:teuthology.orchestra.run.smithi064.stdout:1919 write 0xeda22fd thru 0xedab5c3 (0x92c7 bytes) 2024-04-24T00:34:50.914 INFO:teuthology.orchestra.run.smithi064.stdout:1921 punch from 0x3fdcd36 to 0x3feb4d8, (0xe7a2 bytes) 2024-04-24T00:34:50.914 INFO:teuthology.orchestra.run.smithi064.stdout:1922 read 0x8cfeedc thru 0x8d086cf (0x97f4 bytes) 2024-04-24T00:34:50.914 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:50.914 INFO:teuthology.orchestra.run.smithi064.stdout:1924 write 0x942b93d thru 0x94331bd (0x7881 bytes) 2024-04-24T00:34:50.918 INFO:teuthology.orchestra.run.smithi064.stdout:1925 read 0xbe896bb thru 0xbe92589 (0x8ecf bytes) 2024-04-24T00:34:50.918 INFO:teuthology.orchestra.run.smithi064.stdout:1926 writesame 0xac22fd5 thru 0xac2850e data_size 0x1(0x553a bytes) 2024-04-24T00:34:50.962 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:34:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:34:50.962 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:50.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:50.984 INFO:teuthology.orchestra.run.smithi064.stdout:1928 trunc from 0xedab5c4 to 0x83bb827 2024-04-24T00:34:51.005 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:50.998+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:51.019 INFO:teuthology.orchestra.run.smithi064.stdout:1929 trunc from 0x83bb827 to 0xd4e83ac 2024-04-24T00:34:51.047 INFO:teuthology.orchestra.run.smithi064.stdout:1930 trunc from 0xd4e83ac to 0x6854de4 2024-04-24T00:34:51.060 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:51.060 INFO:teuthology.orchestra.run.smithi064.stdout:1933 read 0x3d56d94 thru 0x3d5de2c (0x7099 bytes) 2024-04-24T00:34:51.060 INFO:teuthology.orchestra.run.smithi064.stdout:1934 punch from 0x3e617e6 to 0x3e63728, (0x1f42 bytes) 2024-04-24T00:34:51.089 INFO:teuthology.orchestra.run.smithi064.stdout:1936 write 0xd97b51a thru 0xd988806 (0xd2ed bytes) 2024-04-24T00:34:51.105 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:51.100+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:51.112 INFO:teuthology.orchestra.run.smithi064.stdout:1938 trunc from 0xd988807 to 0x7a04bc6 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.218052599000004, 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:34:51.148 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:34:51.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:34:51.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:34:51.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:34:51.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4984 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.550708651999997, 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024665092, 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4984 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4e-07 2024-04-24T00:34:51.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593755+0000", 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6829999999999999e-06 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593765+0000", 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7140000000000005e-06 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593773+0000", 2024-04-24T00:34:51.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7829999999999999e-06 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609097+0000", 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015324669000000001 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609254+0000", 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015632499999999999 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609918+0000", 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066479400000000002 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.615678+0000", 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057599119999999998 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618123+0000", 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024450909999999999 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618280+0000", 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156706 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618415+0000", 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134475 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5029 3.f 3:f41ea3ee:::rbd_data.115ed6c33904.0000000000000006:head [zero 262144~262144,zero 786432~65536,zero 917504~39917,write 957421~8409 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:51.160 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.535257287, 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024904274000000001, 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5029 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609208+0000", 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8959999999999997e-06 2024-04-24T00:34:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609285+0000", 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6738999999999998e-05 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609308+0000", 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3896999999999998e-05 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609323+0000", 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4543e-05 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622194+0000", 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012870985 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622559+0000", 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036503000000000002 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.623244+0000", 2024-04-24T00:34:51.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068464100000000005 2024-04-24T00:34:51.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.629548+0000", 2024-04-24T00:34:51.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063044329999999999 2024-04-24T00:34:51.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.633948+0000", 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044003250000000001 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634037+0000", 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9109000000000004e-05 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634105+0000", 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7675999999999994e-05 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5306 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.459121688000003, 2024-04-24T00:34:51.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025510631999999998, 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5306 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685337+0000", 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685342+0000", 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4560000000000002e-06 2024-04-24T00:34:51.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685352+0000", 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840000000000006e-06 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685359+0000", 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7100000000000007e-06 2024-04-24T00:34:51.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696262+0000", 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010902604 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696476+0000", 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021439300000000001 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.697460+0000", 2024-04-24T00:34:51.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00098361900000000003 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.706972+0000", 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095118609999999999 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710681+0000", 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037093619999999999 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710762+0000", 2024-04-24T00:34:51.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0374e-05 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710847+0000", 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5478999999999999e-05 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10360 3.11 3:8fda792a:::rbd_data.115e380d621c.0000000000000177:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:51.173 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.693067720000002, 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042632046999999999, 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10360 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:51.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451393+0000", 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5600000000000001e-06 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451406+0000", 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2984e-05 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451445+0000", 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9045999999999999e-05 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451461+0000", 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5786000000000001e-05 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451608+0000", 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014694900000000001 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451914+0000", 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306657 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002601744 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493753+0000", 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039236409 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493841+0000", 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8448000000000004e-05 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493931+0000", 2024-04-24T00:34:51.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0198999999999995e-05 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494023+0000", 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1265000000000001e-05 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10360 3.11 e40/21 ondisk, result = 0)", 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.682202427, 2024-04-24T00:34:51.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031780810999999999, 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462258+0000", 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2539999999999999e-06 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462266+0000", 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8029999999999997e-06 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462294+0000", 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8637e-05 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462314+0000", 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493814+0000", 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031500133 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493831+0000", 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7357e-05 2024-04-24T00:34:51.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494037+0000", 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020520400000000001 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.286086910999998, 2024-04-24T00:34:51.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:34:51.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:34:51.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:34:51.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.284838163000003, 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10590 3.13 e41/21 ondisk, result = 0)", 2024-04-24T00:34:51.194 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.273592579000002, 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042592342999999998, 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870868+0000", 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1730000000000002e-06 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870876+0000", 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2069999999999995e-06 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870900+0000", 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.404e-05 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870914+0000", 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.366e-05 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913167+0000", 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042252855999999998 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913184+0000", 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7779e-05 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913458+0000", 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027362799999999999 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10592 3.16 e41/21 ondisk, result = 0)", 2024-04-24T00:34:51.198 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.269175982999997, 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044174204000000002, 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:51.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875284+0000", 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1050000000000002e-06 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875292+0000", 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3200000000000002e-06 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875319+0000", 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7382999999999999e-05 2024-04-24T00:34:51.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875334+0000", 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4786e-05 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919179+0000", 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043845151999999998 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919195+0000", 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5846999999999999e-05 2024-04-24T00:34:51.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919456+0000", 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026161100000000003 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.127214059, 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:34:51.202 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:34:51.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:34:51.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:34:51.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.113719516, 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.208 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21553 3.7 3:e2dbd82a:::rbd_data.115eeb9272a5.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.048236411000001, 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029891540000000001, 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.212 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21553 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096223+0000", 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3570000000000001e-06 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096230+0000", 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4080000000000003e-06 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096244+0000", 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4236999999999999e-05 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096251+0000", 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6569999999999997e-06 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096335+0000", 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3997000000000002e-05 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096473+0000", 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013718199999999999 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.097939+0000", 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014665570000000001 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125952+0000", 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028013198 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125977+0000", 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4613999999999998e-05 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126035+0000", 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7966e-05 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126113+0000", 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8367000000000004e-05 2024-04-24T00:34:51.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.851938400999998, 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:34:51.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21613 3.7 e52/44 ondisk, result = 0)", 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.844104462000001, 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044307553999999999, 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:51.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300355+0000", 2024-04-24T00:34:51.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4610000000000001e-06 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300362+0000", 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4930000000000003e-06 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300382+0000", 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9887000000000001e-05 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300394+0000", 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2799000000000001e-05 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344558+0000", 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044164017999999999 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344589+0000", 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.027e-05 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344661+0000", 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2626000000000006e-05 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.598352258, 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.229 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:34:51.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:34:51.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.570678394, 2024-04-24T00:34:51.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:34:51.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.240 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.076655574, 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.241 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:34:51.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:34:51.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.067646740000001, 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30153 3.11 3:8dec0a39:::rbd_data.115e8210e51a.0000000000000015:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.9874643519999999, 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026293440000000001, 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:51.252 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30153 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156998+0000", 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7270000000000001e-06 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157022+0000", 2024-04-24T00:34:51.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3958999999999999e-05 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157065+0000", 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3643000000000002e-05 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157090+0000", 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.446e-05 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157200+0000", 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110406 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157303+0000", 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102545 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159693+0000", 2024-04-24T00:34:51.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023907830000000001 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183138+0000", 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023444415999999999 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183198+0000", 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0331999999999998e-05 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183245+0000", 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7154000000000003e-05 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183287+0000", 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2014999999999997e-05 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:51.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:51.259 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:51.261 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:51.256+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:51.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:51.262 INFO:teuthology.orchestra.run.smithi064.stdout:1939 write 0xaf5b8cd thru 0xaf5c2bb (0x9ef bytes) 2024-04-24T00:34:51.262 INFO:teuthology.orchestra.run.smithi064.stdout:1940 punch from 0x545fd99 to 0x546e9fa, (0xec61 bytes) 2024-04-24T00:34:51.262 INFO:teuthology.orchestra.run.smithi064.stdout:1941 trunc from 0xaf5c2bc to 0x13faf51 2024-04-24T00:34:51.262 INFO:teuthology.orchestra.run.smithi064.stdout:1942 read 0x718647 thru 0x728201 (0xfbbb bytes) 2024-04-24T00:34:51.262 INFO:teuthology.orchestra.run.smithi064.stdout:1944 read 0x11bb51c thru 0x11c7f51 (0xca36 bytes) 2024-04-24T00:34:51.262 INFO:teuthology.orchestra.run.smithi064.stdout:1945 read 0x10b6d3c thru 0x10c0f2e (0xa1f3 bytes) 2024-04-24T00:34:51.263 INFO:teuthology.orchestra.run.smithi064.stdout:1946 trunc from 0x13faf51 to 0x53fc056 2024-04-24T00:34:51.263 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:51.263 INFO:teuthology.orchestra.run.smithi064.stdout:1948 read 0x3fd411 thru 0x40c1cd (0xedbd bytes) 2024-04-24T00:34:51.263 INFO:teuthology.orchestra.run.smithi064.stdout:1949 read 0x1c36079 thru 0x1c38cbe (0x2c46 bytes) 2024-04-24T00:34:51.263 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:51.263 INFO:teuthology.orchestra.run.smithi064.stdout:1952 read 0x3daec7 thru 0x3e2a03 (0x7b3d bytes) 2024-04-24T00:34:51.265 INFO:teuthology.orchestra.run.smithi064.stdout:1954 writesame 0xbb91ff2 thru 0xbb99fa1 data_size 0x48(0x7fb0 bytes) 2024-04-24T00:34:51.265 INFO:teuthology.orchestra.run.smithi064.stdout:1955 read 0x2e26315 thru 0x2e2c2d5 (0x5fc1 bytes) 2024-04-24T00:34:51.278 INFO:teuthology.orchestra.run.smithi064.stdout:1956 write 0xd5a129c thru 0xd5a62c1 (0x5026 bytes) 2024-04-24T00:34:51.297 INFO:teuthology.orchestra.run.smithi064.stdout:1958 write 0xe099cd1 thru 0xe0a722b (0xd55b bytes) 2024-04-24T00:34:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:1959 read 0x1d1da4b thru 0x1d2669c (0x8c52 bytes) 2024-04-24T00:34:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:1960 read 0x854d48e thru 0x855c89c (0xf40f bytes) 2024-04-24T00:34:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:1962 write 0x6fd1d4a thru 0x6fde3d6 (0xc68d bytes) 2024-04-24T00:34:51.315 INFO:teuthology.orchestra.run.smithi064.stdout:1964 write 0x87209ac thru 0x872a961 (0x9fb6 bytes) 2024-04-24T00:34:51.325 INFO:teuthology.orchestra.run.smithi064.stdout:1965 punch from 0xd07f132 to 0xd0831b0, (0x407e bytes) 2024-04-24T00:34:51.325 INFO:teuthology.orchestra.run.smithi064.stdout:1966 writesame 0xbfa7431 thru 0xbfac776 data_size 0x66(0x5346 bytes) 2024-04-24T00:34:51.331 INFO:teuthology.orchestra.run.smithi064.stdout:1967 read 0x5fc7559 thru 0x5fc766a (0x112 bytes) 2024-04-24T00:34:51.331 INFO:teuthology.orchestra.run.smithi064.stdout:1971 write 0x7e481a2 thru 0x7e4f243 (0x70a2 bytes) 2024-04-24T00:34:51.339 INFO:teuthology.orchestra.run.smithi064.stdout:1972 read 0x6b4fbe4 thru 0x6b5d97b (0xdd98 bytes) 2024-04-24T00:34:51.339 INFO:teuthology.orchestra.run.smithi064.stdout:1973 read 0x8cbd094 thru 0x8cc5994 (0x8901 bytes) 2024-04-24T00:34:51.339 INFO:teuthology.orchestra.run.smithi064.stdout:1974 clone 12 order 19 su 2048 sc 6 2024-04-24T00:34:51.361 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:51.356+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:51.396 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:51.396 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:51.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:51.497 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:51.494+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:51.562 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:51.556+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:51.569 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:51.569 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:51.570 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:51.570 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:51.570 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:51.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:34:51.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:51.663 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:51.660+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:51.764 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:51.758+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2047 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263752506000003, 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010295987, 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2047 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530472+0000", 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2799999999999995e-07 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530475+0000", 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0939999999999999e-06 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530485+0000", 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7969999999999995e-06 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530491+0000", 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8709999999999999e-06 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540514+0000", 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010022499000000001 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540626+0000", 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111981 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540767+0000", 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014091700000000001 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2048 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263721695000001, 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010591068, 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2048 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2699999999999999e-07 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530520+0000", 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8165999999999999e-05 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530533+0000", 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2065999999999999e-05 2024-04-24T00:34:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540812+0000", 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010266239 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540925+0000", 2024-04-24T00:34:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112676 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541093+0000", 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016786500000000001 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2049 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263665451999998, 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010815047, 2024-04-24T00:34:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2049 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530562+0000", 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4369999999999999e-06 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530572+0000", 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530579+0000", 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541138+0000", 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010558209000000001 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541239+0000", 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101483 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541373+0000", 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133769 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2050 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263634103000001, 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011052364, 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2050 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530590+0000", 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6599999999999996e-07 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530593+0000", 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.937e-06 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530602+0000", 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4360000000000008e-06 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530609+0000", 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0090000000000001e-06 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541411+0000", 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01080191 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541511+0000", 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9839000000000004e-05 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541642+0000", 2024-04-24T00:34:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130667 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2051 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263604743000002, 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011318002000000001, 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2051 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2900000000000004e-07 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530622+0000", 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9490000000000001e-06 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530632+0000", 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999994e-06 2024-04-24T00:34:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.131e-06 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541683+0000", 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011044629 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120383 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541937+0000", 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132941 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2052 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263575543000002, 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011583872, 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2052 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.44e-07 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530658+0000", 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9850000000000001e-06 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530668+0000", 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530675+0000", 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6240000000000002e-06 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541978+0000", 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011302921 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542100+0000", 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121593 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542232+0000", 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013176600000000001 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2077 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263538142999998, 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011842221, 2024-04-24T00:34:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2077 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6000000000000003e-07 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.095e-06 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530699+0000", 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530706+0000", 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2429999999999999e-06 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542275+0000", 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011568746 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542386+0000", 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011176200000000001 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542527+0000", 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014114499999999999 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2078 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263507816999997, 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012081010999999999, 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2078 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2200000000000004e-07 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530719+0000", 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9349999999999999e-06 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530728+0000", 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0860000000000008e-06 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530735+0000", 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2169999999999997e-06 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542568+0000", 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011832540000000001 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542667+0000", 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9000999999999996e-05 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542797+0000", 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012971 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2079 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263478503999998, 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012334128999999999, 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2079 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-04-24T00:34:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530748+0000", 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9040000000000001e-06 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2159999999999995e-06 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530764+0000", 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9890000000000003e-06 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542839+0000", 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012074354000000001 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542945+0000", 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106443 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543079+0000", 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013369599999999999 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2080 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263449704000003, 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012588825999999999, 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2080 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:34:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530777+0000", 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.022e-06 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9879999999999993e-06 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530793+0000", 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1409999999999999e-06 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543122+0000", 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012328274 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543230+0000", 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108046 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543362+0000", 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013286799999999999 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2081 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263420818999997, 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012843515, 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2081 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1500000000000005e-07 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.819e-06 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530815+0000", 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3540000000000006e-06 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530835+0000", 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9983000000000002e-05 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543405+0000", 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012569739 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543513+0000", 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010821500000000001 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543646+0000", 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289000000000001 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2082 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263377603999999, 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013081859, 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2082 2024-04-24T00:34:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-04-24T00:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530849+0000", 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0750000000000002e-06 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530859+0000", 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7079999999999992e-06 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530866+0000", 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4309999999999997e-06 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543689+0000", 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012822750000000001 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543796+0000", 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106526 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543928+0000", 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131846 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2083 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263347330999999, 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013453964000000001, 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2083 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530877+0000", 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530887+0000", 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186e-05 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530896+0000", 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5480000000000007e-06 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530904+0000", 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4599999999999997e-06 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543970+0000", 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013066067000000001 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544178+0000", 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020785000000000001 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544330+0000", 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015230699999999999 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2084 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.263309507000002, 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907047000000001, 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2084 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3099999999999998e-07 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530922+0000", 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.498e-06 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530932+0000", 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530940+0000", 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4769999999999997e-06 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544421+0000", 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013481739 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544673+0000", 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025143600000000001 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544821+0000", 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014819 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.360232142000001, 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:34:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:34:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.304720656000001, 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:34:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:34:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:34:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:34:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33687 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.4112409619999999, 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015720043, 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33687 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382984+0000", 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382994+0000", 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.008e-05 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383020+0000", 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5565999999999999e-05 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383034+0000", 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:34:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383167+0000", 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132719 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383325+0000", 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015864899999999999 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.384182+0000", 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085650199999999998 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398477+0000", 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014294836 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398509+0000", 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2731999999999998e-05 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398615+0000", 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106084 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398702+0000", 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7068000000000001e-05 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33687 3.6 e70/59 ondisk, result = 0)", 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.4067800259999999, 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011051082, 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387445+0000", 2024-04-24T00:34:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387455+0000", 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387483+0000", 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8181999999999999e-05 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387497+0000", 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.401e-05 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398435+0000", 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010938296 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398464+0000", 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8881999999999999e-05 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398494+0000", 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0219e-05 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2041440080000001, 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:34:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:34:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1625790540000001, 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:34:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:34:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:51.884 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:51.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:51.886 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:51.884+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:51.987 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:51.984+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:52.053 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:52.053 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:52.053 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:52.053 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:52.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:52.087 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:52.084+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:52.184 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:52.184 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:52.184 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:52.188 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:52.184+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:52.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:52.290 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:52.284+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:52.329 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:52.329 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2979 3.9 e33/21 3:9717bf35:::rbd_data.115e5ff16b0c.000000000000001b:head v 33'28, mlcod=30'26)", 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.503145263, 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053184019999999999, 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:52.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:52.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822203+0000", 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3909999999999999e-06 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822482+0000", 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027925399999999999 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822509+0000", 2024-04-24T00:34:52.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.707e-05 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822521+0000", 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1779e-05 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822626+0000", 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105165 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822653+0000", 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6713e-05 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827477+0000", 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048240949999999996 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827520+0000", 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2935000000000001e-05 2024-04-24T00:34:52.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.680540630000003, 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10425 3.4 e41/21 3:2657577c:::rbd_data.115e380d621c.0000000000000210:head v 41'28, mlcod=39'26)", 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.707069955999998, 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041008809999999998, 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:52.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618285+0000", 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5580000000000003e-06 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618309+0000", 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-05 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618321+0000", 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.164e-05 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619328+0000", 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010071259999999999 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619360+0000", 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1581000000000001e-05 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622325+0000", 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029648610000000001 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622378+0000", 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3235000000000001e-05 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10705 3.9 e41/21 3:93f8bbf0:::rbd_data.115e380d621c.0000000000000399:head v 41'63, mlcod=41'61)", 2024-04-24T00:34:52.343 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.163109931000001, 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034786589999999998, 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162238+0000", 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162284+0000", 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6122999999999999e-05 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162296+0000", 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1871e-05 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162304+0000", 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7030000000000006e-06 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162390+0000", 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6582999999999994e-05 2024-04-24T00:34:52.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162431+0000", 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0577999999999997e-05 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165655+0000", 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003224612 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165716+0000", 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0251999999999998e-05 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20211 3.4 e48/21 3:249c5675:::rbd_data.115ee4742374.0000000000000021:head v 48'33, mlcod=41'31)", 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.044350152, 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004558655, 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280999+0000", 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0559999999999999e-06 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281071+0000", 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2300999999999998e-05 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281102+0000", 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1078000000000003e-05 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281116+0000", 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4273e-05 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281198+0000", 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1364000000000002e-05 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281221+0000", 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3589999999999999e-05 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.285420+0000", 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041987559999999997 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.285555+0000", 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013523699999999999 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20552 3.9 e49/21 3:923662ec:::rbd_data.115ee4742374.000000000000000a:head v 49'88, mlcod=48'86)", 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.351847029999998, 2024-04-24T00:34:52.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062734230000000002, 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973502+0000", 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.04e-06 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973677+0000", 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473200000000001 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973714+0000", 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7117999999999999e-05 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973734+0000", 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0102000000000002e-05 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973879+0000", 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145654 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973946+0000", 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6476999999999994e-05 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979687+0000", 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057411140000000003 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979773+0000", 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6186000000000005e-05 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20615 3.1f e49/21 3:fddc1920:::rbd_data.115ee4742374.0000000000000018:head v 49'81, mlcod=49'79)", 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.089548179000001, 2024-04-24T00:34:52.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045618899999999999, 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235801+0000", 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235887+0000", 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5688999999999999e-05 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235922+0000", 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5719000000000001e-05 2024-04-24T00:34:52.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235941+0000", 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8916e-05 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.236065+0000", 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012414699999999999 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.236105+0000", 2024-04-24T00:34:52.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0091e-05 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.240269+0000", 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041637339999999997 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.240361+0000", 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1310000000000005e-05 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.742265774, 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:52.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:34:52.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:34:52.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.742191919, 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:34:52.366 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.370 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.045497135000002, 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:34:52.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:34:52.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:22507 3.1f e63/59 3:ff4fbbf8:::rbd_data.115e6a380959.000000000000000c:head v 63'86, mlcod=63'84)", 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.564955012, 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052552429999999997, 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760396+0000", 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7290000000000002e-06 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760534+0000", 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138543 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760591+0000", 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7321000000000002e-05 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760612+0000", 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0239999999999999e-05 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760775+0000", 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016357000000000001 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760819+0000", 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3843e-05 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.765551+0000", 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047318270000000001 2024-04-24T00:34:52.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.765647+0000", 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6169999999999996e-05 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33705 3.1f e70/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 70'124, mlcod=70'122)", 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.8741289820000002, 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035074059999999998, 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.379 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451220+0000", 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0700000000000001e-06 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451266+0000", 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6059000000000001e-05 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451294+0000", 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8339000000000001e-05 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451311+0000", 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7076e-05 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451443+0000", 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013130999999999999 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451486+0000", 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3015000000000001e-05 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.454629+0000", 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003143046 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.454725+0000", 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6490999999999995e-05 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33828 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'128, mlcod=71'126)", 2024-04-24T00:34:52.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.3815119610000002, 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00348315, 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943837+0000", 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6230000000000001e-06 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943901+0000", 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3255000000000003e-05 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943936+0000", 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5191999999999998e-05 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943952+0000", 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6439999999999998e-05 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.944112+0000", 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000159946 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.944156+0000", 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3455000000000003e-05 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.947168+0000", 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030124259999999999 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.947318+0000", 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149813 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.387 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33843 3.0 e71/21 3:04e211c8:::rbd_data.115ec4f9b3a7.000000000000000f:head v 71'81, mlcod=69'79)", 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.3457033709999999, 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041071550000000004, 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979645+0000", 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.302e-06 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979667+0000", 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1821e-05 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979684+0000", 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7856999999999999e-05 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979694+0000", 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0049e-05 2024-04-24T00:34:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979827+0000", 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013208800000000001 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979868+0000", 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1066000000000003e-05 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.983684+0000", 2024-04-24T00:34:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038166749999999998 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.983751+0000", 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6297000000000006e-05 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2399651760000001, 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:34:52.392 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:34:52.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:34:52.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33935 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'135, mlcod=71'133)", 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1035745649999997, 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052923950000000001, 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221773+0000", 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0109999999999999e-06 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221814+0000", 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0547999999999999e-05 2024-04-24T00:34:52.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221829+0000", 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221839+0000", 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9399999999999997e-06 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221980+0000", 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014113999999999999 2024-04-24T00:34:52.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.222046+0000", 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6107999999999998e-05 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.226980+0000", 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049344890000000002 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.227065+0000", 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4258000000000005e-05 2024-04-24T00:34:52.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:35459 3.4 e74/21 3:26775e64:::rbd_object_map.115ec5833ea9:head v 74'96, mlcod=74'92)", 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.9404378289999999, 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043560150000000004, 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384910+0000", 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384917+0000", 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3599999999999998e-06 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384933+0000", 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5821e-05 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384942+0000", 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8769999999999999e-06 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385004+0000", 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2359000000000004e-05 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385027+0000", 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2878999999999999e-05 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389218+0000", 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004190399 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389265+0000", 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7445999999999999e-05 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.1f epoch 74/59 rep_tid 451 entries 74'138 (0'0) error 3:f8463f1b:::rbd_data.115ec5833ea9.0000000000000056:head by client.4446.0:35462 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 73'136)", 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.940206576, 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041792909999999999, 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385141+0000", 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1600000000000002e-07 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385147+0000", 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9419999999999998e-06 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385170+0000", 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2966999999999999e-05 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385186+0000", 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5424999999999999e-05 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385266+0000", 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0408999999999995e-05 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389320+0000", 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040532320000000004 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.0 epoch 74/21 rep_tid 368 entries 74'86 (0'0) error 3:0193a4e3:::rbd_data.115ec5833ea9.0000000000000057:head by client.4446.0:35463 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 74'84)", 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.9395930539999999, 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035511779999999999, 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385755+0000", 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4099999999999997e-07 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385766+0000", 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1070000000000001e-05 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385786+0000", 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0492e-05 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385798+0000", 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1714e-05 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385883+0000", 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5068000000000006e-05 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389305+0000", 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034219929999999999 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:35490 3.9 e74/21 3:93eb3546:::rbd_data.115ec5833ea9.0000000000000020:head v 74'198, mlcod=74'196)", 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.8956501029999999, 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038708850000000001, 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:52.412 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429698+0000", 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0550000000000001e-06 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429742+0000", 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4011999999999998e-05 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429755+0000", 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3664e-05 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429762+0000", 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2210000000000002e-06 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429827+0000", 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5584000000000001e-05 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429853+0000", 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5817e-05 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.433515+0000", 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036616980000000001 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:52.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:52.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.433568+0000", 2024-04-24T00:34:52.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2834000000000003e-05 2024-04-24T00:34:52.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:52.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:52.416 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:52.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:52.418 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:52.412+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:52.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:52.518 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:52.516+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:52.548 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:52.548 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:52.548 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:52.548 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:52.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:52.607 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:34:52.619 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:52.614+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:52.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:52.627 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:52.641 INFO:tasks.thrashosds.thrasher:No PGs found for osd.0 2024-04-24T00:34:52.645 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:52.645 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:52.662 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:34:52.665 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:52.665 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:52.695 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:52.695 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:52.701 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:52.702 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:52.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:52.721 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:52.716+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:52.736 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:52.736 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:52.761 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:52.761 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:52.786 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:52.787 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:34:52.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:34:52.833 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:52.830+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:52.869 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:52.870 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:52.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:52.930 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:52.928+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:53.024 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:53.018+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:53.064 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:53.064 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:53.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:53.130 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:53.128+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:53.227 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:53.224+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:53.282 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:53.282 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:53.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:53.327 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:53.322+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:53.429 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:53.424+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:53.476 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:53.477 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:53.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:53.535 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:53.532+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:53.631 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:53.628+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:53.657 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:53.657 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:53.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:53.732 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:53.726+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:53.833 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:53.828+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:53.852 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:53.852 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:53.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:53.910 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-04-24T00:34:53.963 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:53.960+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:54.034 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:54.028+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:54.043 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:54.043 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:54.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:54.146 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:54.144+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:54.218 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:54.218 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:54.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:54.274 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:54.272+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:54.299 INFO:teuthology.orchestra.run.smithi064.stdout:1975 writesame 0xe6f6497 thru 0xe6f8203 data_size 0x1f(0x1d6d bytes) 2024-04-24T00:34:54.299 INFO:teuthology.orchestra.run.smithi064.stdout:1977 read 0xe50ead7 thru 0xe50eb8e (0xb8 bytes) 2024-04-24T00:34:54.299 INFO:teuthology.orchestra.run.smithi064.stdout:1978 read 0x49d1d61 thru 0x49dd4dc (0xb77c bytes) 2024-04-24T00:34:54.312 INFO:teuthology.orchestra.run.smithi064.stdout:1979 trunc from 0xe6f8204 to 0x5a6b932 2024-04-24T00:34:54.336 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:54.330+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:54.353 INFO:teuthology.orchestra.run.smithi064.stdout:1980 read 0x3272de8 thru 0x327b2e1 (0x84fa bytes) 2024-04-24T00:34:54.353 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:54.353 INFO:teuthology.orchestra.run.smithi064.stdout:1982 read 0x30955bd thru 0x309b736 (0x617a bytes) 2024-04-24T00:34:54.364 INFO:teuthology.orchestra.run.smithi064.stdout:1983 write 0x5cdbdd5 thru 0x5ce2bd3 (0x6dff bytes) 2024-04-24T00:34:54.394 INFO:teuthology.orchestra.run.smithi064.stdout:1984 trunc from 0x5ce2bd4 to 0xb132c3 2024-04-24T00:34:54.417 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:54.417 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:54.432 INFO:teuthology.orchestra.run.smithi064.stdout:1985 trunc from 0xb132c3 to 0x1f473ed 2024-04-24T00:34:54.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:1987 writesame 0x6d8cfe4 thru 0x6d8ecc0 data_size 0x3(0x1cdd bytes) 2024-04-24T00:34:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:1988 punch from 0x10f1c57 to 0x10f9b3f, (0x7ee8 bytes) 2024-04-24T00:34:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:1989 write 0x3eb6e62 thru 0x3ebf40e (0x85ad bytes) 2024-04-24T00:34:54.478 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:54.476+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:54.481 INFO:teuthology.orchestra.run.smithi064.stdout:1990 punch from 0x4e402e2 to 0x4e44f23, (0x4c41 bytes) 2024-04-24T00:34:54.481 INFO:teuthology.orchestra.run.smithi064.stdout:1991 read 0x4ab7ff6 thru 0x4ab9faa (0x1fb5 bytes) 2024-04-24T00:34:54.481 INFO:teuthology.orchestra.run.smithi064.stdout:1992 punch from 0x200ab41 to 0x201912b, (0xe5ea bytes) 2024-04-24T00:34:54.481 INFO:teuthology.orchestra.run.smithi064.stdout:1993 read 0x40be342 thru 0x40c2fa6 (0x4c65 bytes) 2024-04-24T00:34:54.481 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:54.482 INFO:teuthology.orchestra.run.smithi064.stdout:1996 punch from 0x477015c to 0x47781ae, (0x8052 bytes) 2024-04-24T00:34:54.506 INFO:teuthology.orchestra.run.smithi064.stdout:1997 write 0xbece438 thru 0xbeda374 (0xbf3d bytes) 2024-04-24T00:34:54.517 INFO:teuthology.orchestra.run.smithi064.stdout:1998 read 0x359b5d1 thru 0x359daa8 (0x24d8 bytes) 2024-04-24T00:34:54.538 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:54.532+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:54.543 INFO:teuthology.orchestra.run.smithi064.stdout:1999 trunc from 0xbeda375 to 0x141e772 2024-04-24T00:34:54.567 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:54.567 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:54.571 INFO:teuthology.orchestra.run.smithi064.stdout:2000 punch from 0x120b0ca to 0x1213423, (0x8359 bytes) 2024-04-24T00:34:54.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:54.612 INFO:teuthology.orchestra.run.smithi064.stdout:2002 write 0xc919daf thru 0xc921c5f (0x7eb1 bytes) 2024-04-24T00:34:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:2003 read 0x84b1f86 thru 0x84bedfb (0xce76 bytes) 2024-04-24T00:34:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:2006 read 0xb4a3cc9 thru 0xb4a615d (0x2495 bytes) 2024-04-24T00:34:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:2008 punch from 0xa390658 to 0xa39a6f7, (0xa09f bytes) 2024-04-24T00:34:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:2009 read 0x975657a thru 0x97660ba (0xfb41 bytes) 2024-04-24T00:34:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:2010 read 0xa40cb47 thru 0xa41c858 (0xfd12 bytes) 2024-04-24T00:34:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:2011 read 0x62b0d29 thru 0x62b6396 (0x566e bytes) 2024-04-24T00:34:54.640 INFO:teuthology.orchestra.run.smithi064.stdout:2012 trunc from 0xc921c60 to 0x6b3f8d8 2024-04-24T00:34:54.666 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:54.664+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:54.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:34:54.689 INFO:teuthology.orchestra.run.smithi064.stdout:2013 writesame 0xc53d2b0 thru 0xc54c0c2 data_size 0x1(0xee13 bytes) 2024-04-24T00:34:54.689 INFO:teuthology.orchestra.run.smithi064.stdout:2014 punch from 0x91abca7 to 0x91b1be6, (0x5f3f bytes) 2024-04-24T00:34:54.689 INFO:teuthology.orchestra.run.smithi064.stdout:2015 read 0x6df36e7 thru 0x6df3adf (0x3f9 bytes) 2024-04-24T00:34:54.689 INFO:teuthology.orchestra.run.smithi064.stdout:2016 writesame 0x358086d thru 0x3581489 data_size 0x1(0xc1d bytes) 2024-04-24T00:34:54.711 INFO:teuthology.orchestra.run.smithi064.stdout:2018 trunc from 0xc54c0c3 to 0x63cea59 2024-04-24T00:34:54.721 INFO:teuthology.orchestra.run.smithi064.stdout:2019 read 0x34de608 thru 0x34e623d (0x7c36 bytes) 2024-04-24T00:34:54.722 INFO:teuthology.orchestra.run.smithi064.stdout:2020 punch from 0x2dc4374 to 0x2dcfe1b, (0xbaa7 bytes) 2024-04-24T00:34:54.722 INFO:teuthology.orchestra.run.smithi064.stdout:2021 trunc from 0x63cea59 to 0x61104dd 2024-04-24T00:34:54.731 INFO:teuthology.orchestra.run.smithi064.stdout:2022 read 0x506d447 thru 0x50765bc (0x9176 bytes) 2024-04-24T00:34:54.739 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:54.736+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:54.740 INFO:teuthology.orchestra.run.smithi064.stdout:2023 write 0x665378b thru 0x665b405 (0x7c7b bytes) 2024-04-24T00:34:54.754 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:54.754 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:54.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:54.784 INFO:teuthology.orchestra.run.smithi064.stdout:2024 write 0xcbf3355 thru 0xcbf6d75 (0x3a21 bytes) 2024-04-24T00:34:54.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:54.796 INFO:teuthology.orchestra.run.smithi064.stdout:2025 write 0x90d6d27 thru 0x90e6baa (0xfe84 bytes) 2024-04-24T00:34:54.808 INFO:teuthology.orchestra.run.smithi064.stdout:2026 read 0x26c4483 thru 0x26d138f (0xcf0d bytes) 2024-04-24T00:34:54.808 INFO:teuthology.orchestra.run.smithi064.stdout:2027 punch from 0x2d5b410 to 0x2d640c2, (0x8cb2 bytes) 2024-04-24T00:34:54.808 INFO:teuthology.orchestra.run.smithi064.stdout:2028 punch from 0x7561e2f to 0x7567839, (0x5a0a bytes) 2024-04-24T00:34:54.808 INFO:teuthology.orchestra.run.smithi064.stdout:2029 read 0x1236e06 thru 0x123d6d8 (0x68d3 bytes) 2024-04-24T00:34:54.808 INFO:teuthology.orchestra.run.smithi064.stdout:2032 write 0x9c681ee thru 0x9c688d1 (0x6e4 bytes) 2024-04-24T00:34:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:2033 write 0x461d497 thru 0x461e934 (0x149e bytes) 2024-04-24T00:34:54.827 INFO:teuthology.orchestra.run.smithi064.stdout:2034 writesame 0xa22b86b thru 0xa238cc3 data_size 0x1(0xd459 bytes) 2024-04-24T00:34:54.858 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:54.856+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:54.864 INFO:teuthology.orchestra.run.smithi064.stdout:2035 punch from 0x67cd75e to 0x67d9b4d, (0xc3ef bytes) 2024-04-24T00:34:54.864 INFO:teuthology.orchestra.run.smithi064.stdout:2036 write 0x62e59b0 thru 0x62f57d6 (0xfe27 bytes) 2024-04-24T00:34:54.880 INFO:teuthology.orchestra.run.smithi064.stdout:2037 clone 13 order 24 su 131072 sc 5 2024-04-24T00:34:54.932 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:34:54.934 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:54.934 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:54.941 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:54.934+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:54.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:54.951 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:54.951 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:54.969 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:54.969 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:54.986 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:54.986 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:55.005 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:55.005 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:55.022 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:55.022 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:55.041 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:55.041 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:55.042 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:55.040+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:55.060 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:55.060 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:34:55.066 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.189448041000006, 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.121 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.127 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:4984 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.522104093999999, 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024665092, 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 4984 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593750+0000", 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4e-07 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593755+0000", 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6829999999999999e-06 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593765+0000", 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7140000000000005e-06 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.593773+0000", 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7829999999999999e-06 2024-04-24T00:34:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609097+0000", 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015324669000000001 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609254+0000", 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015632499999999999 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609918+0000", 2024-04-24T00:34:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066479400000000002 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.615678+0000", 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057599119999999998 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618123+0000", 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024450909999999999 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618280+0000", 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156706 2024-04-24T00:34:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.618415+0000", 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134475 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5029 3.f 3:f41ea3ee:::rbd_data.115ed6c33904.0000000000000006:head [zero 262144~262144,zero 786432~65536,zero 917504~39917,write 957421~8409 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.506652729000002, 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024904274000000001, 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5029 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609201+0000", 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609208+0000", 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8959999999999997e-06 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609285+0000", 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6738999999999998e-05 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609308+0000", 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3896999999999998e-05 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.609323+0000", 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4543e-05 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622194+0000", 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012870985 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.622559+0000", 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036503000000000002 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.623244+0000", 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068464100000000005 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.629548+0000", 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063044329999999999 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.633948+0000", 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044003250000000001 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634037+0000", 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9109000000000004e-05 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.634105+0000", 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7675999999999994e-05 2024-04-24T00:34:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5306 3.f 3:f19252c9:::rbd_object_map.115ed6c33904.0000000000000008:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.430517129999998, 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025510631999999998, 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5306 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685336+0000", 2024-04-24T00:34:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685337+0000", 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685342+0000", 2024-04-24T00:34:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4560000000000002e-06 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685352+0000", 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840000000000006e-06 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.685359+0000", 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7100000000000007e-06 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696262+0000", 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010902604 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.696476+0000", 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021439300000000001 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.697460+0000", 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00098361900000000003 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.706972+0000", 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095118609999999999 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710681+0000", 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037093619999999999 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710762+0000", 2024-04-24T00:34:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0374e-05 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.710847+0000", 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5478999999999999e-05 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10360 3.11 3:8fda792a:::rbd_data.115e380d621c.0000000000000177:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.664463161999997, 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042632046999999999, 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10360 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451390+0000", 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451393+0000", 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5600000000000001e-06 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451406+0000", 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2984e-05 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451445+0000", 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9045999999999999e-05 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451461+0000", 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5786000000000001e-05 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451608+0000", 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014694900000000001 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.451914+0000", 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306657 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.454516+0000", 2024-04-24T00:34:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002601744 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493753+0000", 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039236409 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493841+0000", 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8448000000000004e-05 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493931+0000", 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0198999999999995e-05 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494023+0000", 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1265000000000001e-05 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10360 3.11 e40/21 ondisk, result = 0)", 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.653597869000002, 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031780810999999999, 2024-04-24T00:34:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462256+0000", 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462258+0000", 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2539999999999999e-06 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462266+0000", 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8029999999999997e-06 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462294+0000", 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8637e-05 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.462314+0000", 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493814+0000", 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031500133 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.493831+0000", 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7357e-05 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.494037+0000", 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020520400000000001 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.257482353, 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:34:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:34:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.256233604999998, 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:34:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:34:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:34:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:34:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:34:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10590 3.13 e41/21 ondisk, result = 0)", 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.244988020999998, 2024-04-24T00:34:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042592342999999998, 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870866+0000", 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870868+0000", 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1730000000000002e-06 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870876+0000", 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2069999999999995e-06 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870900+0000", 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.404e-05 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.870914+0000", 2024-04-24T00:34:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.366e-05 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913167+0000", 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042252855999999998 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913184+0000", 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7779e-05 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913458+0000", 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027362799999999999 2024-04-24T00:34:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10592 3.16 e41/21 ondisk, result = 0)", 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.240571424999999, 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044174204000000002, 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875282+0000", 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875284+0000", 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1050000000000002e-06 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875292+0000", 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3200000000000002e-06 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875319+0000", 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7382999999999999e-05 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.875334+0000", 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4786e-05 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919179+0000", 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043845151999999998 2024-04-24T00:34:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919195+0000", 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5846999999999999e-05 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919456+0000", 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026161100000000003 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:34:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.098609500999999, 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:34:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.085114957999998, 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:34:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21553 3.7 3:e2dbd82a:::rbd_data.115eeb9272a5.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.019631853, 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029891540000000001, 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21553 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096222+0000", 2024-04-24T00:34:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096223+0000", 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3570000000000001e-06 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096230+0000", 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4080000000000003e-06 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096244+0000", 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4236999999999999e-05 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096251+0000", 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6569999999999997e-06 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096335+0000", 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3997000000000002e-05 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.096473+0000", 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013718199999999999 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.097939+0000", 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014665570000000001 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125952+0000", 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028013198 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.125977+0000", 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4613999999999998e-05 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126035+0000", 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7966e-05 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.126113+0000", 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8367000000000004e-05 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.823333843, 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21613 3.7 e52/44 ondisk, result = 0)", 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.815499903999999, 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044307553999999999, 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300354+0000", 2024-04-24T00:34:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300355+0000", 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4610000000000001e-06 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300362+0000", 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4930000000000003e-06 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300382+0000", 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9887000000000001e-05 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.300394+0000", 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2799000000000001e-05 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344558+0000", 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044164017999999999 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344589+0000", 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.027e-05 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344661+0000", 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2626000000000006e-05 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.569747700000001, 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:34:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.542073836, 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:34:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.048051016000001, 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:34:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:34:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:34:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.039042181999999, 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:34:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30153 3.11 3:8dec0a39:::rbd_data.115e8210e51a.0000000000000015:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.958859794, 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026293440000000001, 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:34:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30153 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156994+0000", 2024-04-24T00:34:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.156998+0000", 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7270000000000001e-06 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157022+0000", 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3958999999999999e-05 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157065+0000", 2024-04-24T00:34:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3643000000000002e-05 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157090+0000", 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.446e-05 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157200+0000", 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110406 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157303+0000", 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102545 2024-04-24T00:34:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159693+0000", 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023907830000000001 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183138+0000", 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023444415999999999 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183198+0000", 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0331999999999998e-05 2024-04-24T00:34:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183245+0000", 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7154000000000003e-05 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.183287+0000", 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2014999999999997e-05 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:34:55.223 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:55.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:55.226 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:55.222+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:55.326 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:55.322+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:55.356 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:55.356 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:55.357 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:55.357 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:55.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:55.427 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:55.423+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:55.528 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:55.522+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:55.532 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:55.532 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:55.546 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:55.629 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:55.623+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:55.695 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:34:55.723 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:55.723 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:55.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:55.723 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:55.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.723 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2047 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.190135777999998, 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010295987, 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2047 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530471+0000", 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530472+0000", 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2799999999999995e-07 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530475+0000", 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0939999999999999e-06 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530485+0000", 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7969999999999995e-06 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530491+0000", 2024-04-24T00:34:55.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8709999999999999e-06 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540514+0000", 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010022499000000001 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540626+0000", 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111981 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540767+0000", 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014091700000000001 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2048 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.190104967000003, 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010591068, 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2048 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530502+0000", 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2699999999999999e-07 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530520+0000", 2024-04-24T00:34:55.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8165999999999999e-05 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530533+0000", 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2065999999999999e-05 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530546+0000", 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3429e-05 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540812+0000", 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010266239 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.540925+0000", 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112676 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541093+0000", 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016786500000000001 2024-04-24T00:34:55.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2049 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.190048724, 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010815047, 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2049 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530558+0000", 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530562+0000", 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4369999999999999e-06 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530572+0000", 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-04-24T00:34:55.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530579+0000", 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541138+0000", 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010558209000000001 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541239+0000", 2024-04-24T00:34:55.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101483 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541373+0000", 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133769 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2050 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.190017374999996, 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011052364, 2024-04-24T00:34:55.737 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2050 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530589+0000", 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530590+0000", 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6599999999999996e-07 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530593+0000", 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.937e-06 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530602+0000", 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4360000000000008e-06 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530609+0000", 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0090000000000001e-06 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541411+0000", 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01080191 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541511+0000", 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9839000000000004e-05 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541642+0000", 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130667 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2051 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:55.741 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189988014999997, 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011318002000000001, 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2051 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530619+0000", 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2900000000000004e-07 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530622+0000", 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9490000000000001e-06 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530632+0000", 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999994e-06 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530639+0000", 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.131e-06 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541683+0000", 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011044629 2024-04-24T00:34:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541804+0000", 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120383 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541937+0000", 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132941 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2052 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189958814999997, 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011583872, 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.746 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2052 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530648+0000", 2024-04-24T00:34:55.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.44e-07 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530658+0000", 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9850000000000001e-06 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530668+0000", 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530675+0000", 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6240000000000002e-06 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.541978+0000", 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011302921 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542100+0000", 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121593 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542232+0000", 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013176600000000001 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2077 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189921415000001, 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011842221, 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.750 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2077 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530685+0000", 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530686+0000", 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6000000000000003e-07 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530689+0000", 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.095e-06 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530699+0000", 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530706+0000", 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2429999999999999e-06 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542275+0000", 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011568746 2024-04-24T00:34:55.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542386+0000", 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011176200000000001 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542527+0000", 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014114499999999999 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2078 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189891089, 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012081010999999999, 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2078 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530716+0000", 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2200000000000004e-07 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530719+0000", 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9349999999999999e-06 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530728+0000", 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0860000000000008e-06 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530735+0000", 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2169999999999997e-06 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542568+0000", 2024-04-24T00:34:55.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011832540000000001 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542667+0000", 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9000999999999996e-05 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542797+0000", 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012971 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2079 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189861776000001, 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012334128999999999, 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2079 2024-04-24T00:34:55.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530745+0000", 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530748+0000", 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9040000000000001e-06 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530758+0000", 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2159999999999995e-06 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530764+0000", 2024-04-24T00:34:55.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9890000000000003e-06 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542839+0000", 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012074354000000001 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.542945+0000", 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106443 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543079+0000", 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013369599999999999 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2080 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189832975999998, 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012588825999999999, 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.763 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2080 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530774+0000", 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530777+0000", 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.022e-06 2024-04-24T00:34:55.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530786+0000", 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9879999999999993e-06 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530793+0000", 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1409999999999999e-06 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543122+0000", 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012328274 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543230+0000", 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108046 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543362+0000", 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013286799999999999 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2081 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189804090999999, 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012843515, 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2081 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530803+0000", 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1500000000000005e-07 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530806+0000", 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.819e-06 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530815+0000", 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3540000000000006e-06 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530835+0000", 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9983000000000002e-05 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543405+0000", 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012569739 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543513+0000", 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010821500000000001 2024-04-24T00:34:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543646+0000", 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289000000000001 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2082 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189760876000001, 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013081859, 2024-04-24T00:34:55.772 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2082 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530846+0000", 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2300000000000001e-07 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530849+0000", 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0750000000000002e-06 2024-04-24T00:34:55.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530859+0000", 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7079999999999992e-06 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530866+0000", 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4309999999999997e-06 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543689+0000", 2024-04-24T00:34:55.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012822750000000001 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543796+0000", 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106526 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543928+0000", 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131846 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2083 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189730603000001, 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013453964000000001, 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2083 2024-04-24T00:34:55.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530876+0000", 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530877+0000", 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530887+0000", 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186e-05 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530896+0000", 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5480000000000007e-06 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530904+0000", 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4599999999999997e-06 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.543970+0000", 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013066067000000001 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544178+0000", 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020785000000000001 2024-04-24T00:34:55.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544330+0000", 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015230699999999999 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2084 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.189692778999998, 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907047000000001, 2024-04-24T00:34:55.781 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2084 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3099999999999998e-07 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530922+0000", 2024-04-24T00:34:55.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.498e-06 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530932+0000", 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530940+0000", 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4769999999999997e-06 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544421+0000", 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013481739 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544673+0000", 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025143600000000001 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544821+0000", 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014819 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.286615414, 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.786 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:34:55.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:34:55.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.231103928, 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33687 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3376242339999997, 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015720043, 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33687 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382984+0000", 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382994+0000", 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.008e-05 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383020+0000", 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5565999999999999e-05 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383034+0000", 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383167+0000", 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132719 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383325+0000", 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015864899999999999 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.384182+0000", 2024-04-24T00:34:55.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085650199999999998 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398477+0000", 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014294836 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398509+0000", 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2731999999999998e-05 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398615+0000", 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106084 2024-04-24T00:34:55.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398702+0000", 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7068000000000001e-05 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33687 3.6 e70/59 ondisk, result = 0)", 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3331632980000006, 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011051082, 2024-04-24T00:34:55.801 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387443+0000", 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387445+0000", 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387455+0000", 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387483+0000", 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8181999999999999e-05 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.387497+0000", 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.401e-05 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398435+0000", 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010938296 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398464+0000", 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8881999999999999e-05 2024-04-24T00:34:55.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398494+0000", 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0219e-05 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1305272800000008, 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:34:55.805 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:34:55.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:34:55.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0889623260000008, 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:55.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:55.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:55.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:55.815 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:55.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:55.817 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:55.811+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:55.917 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:55.914+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:55.990 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:55.990 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:55.990 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:55.990 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:56.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:56.018 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:56.015+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:56.120 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:56.115+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:56.165 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:56.165 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:56.165 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:56.165 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:56.165 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:56.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:34:56.220 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:56.215+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:56.308 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2979 3.9 e33/21 3:9717bf35:::rbd_data.115e5ff16b0c.000000000000001b:head v 33'28, mlcod=30'26)", 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.483739993999997, 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053184019999999999, 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.309 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822203+0000", 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3909999999999999e-06 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822482+0000", 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027925399999999999 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822509+0000", 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.707e-05 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822521+0000", 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1779e-05 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822626+0000", 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105165 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822653+0000", 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6713e-05 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827477+0000", 2024-04-24T00:34:56.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048240949999999996 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827520+0000", 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2935000000000001e-05 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.661135360999999, 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:34:56.313 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:34:56.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:34:56.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10425 3.4 e41/21 3:2657577c:::rbd_data.115e380d621c.0000000000000210:head v 41'28, mlcod=39'26)", 2024-04-24T00:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.687664687000002, 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041008809999999998, 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618285+0000", 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5580000000000003e-06 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618309+0000", 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-05 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618321+0000", 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.164e-05 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619328+0000", 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010071259999999999 2024-04-24T00:34:56.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619360+0000", 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1581000000000001e-05 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622325+0000", 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029648610000000001 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622378+0000", 2024-04-24T00:34:56.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3235000000000001e-05 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10705 3.9 e41/21 3:93f8bbf0:::rbd_data.115e380d621c.0000000000000399:head v 41'63, mlcod=41'61)", 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.143704661999998, 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034786589999999998, 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162238+0000", 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-04-24T00:34:56.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162284+0000", 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6122999999999999e-05 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162296+0000", 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1871e-05 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162304+0000", 2024-04-24T00:34:56.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7030000000000006e-06 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162390+0000", 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6582999999999994e-05 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162431+0000", 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0577999999999997e-05 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165655+0000", 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003224612 2024-04-24T00:34:56.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165716+0000", 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0251999999999998e-05 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20211 3.4 e48/21 3:249c5675:::rbd_data.115ee4742374.0000000000000021:head v 48'33, mlcod=41'31)", 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.024944883000003, 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004558655, 2024-04-24T00:34:56.326 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280999+0000", 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0559999999999999e-06 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281071+0000", 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2300999999999998e-05 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281102+0000", 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1078000000000003e-05 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281116+0000", 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4273e-05 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281198+0000", 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1364000000000002e-05 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281221+0000", 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3589999999999999e-05 2024-04-24T00:34:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.285420+0000", 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041987559999999997 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.285555+0000", 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013523699999999999 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20552 3.9 e49/21 3:923662ec:::rbd_data.115ee4742374.000000000000000a:head v 49'88, mlcod=48'86)", 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.332441760999998, 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062734230000000002, 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973502+0000", 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.04e-06 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973677+0000", 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473200000000001 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973714+0000", 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7117999999999999e-05 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973734+0000", 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0102000000000002e-05 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973879+0000", 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145654 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973946+0000", 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6476999999999994e-05 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979687+0000", 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057411140000000003 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979773+0000", 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6186000000000005e-05 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20615 3.1f e49/21 3:fddc1920:::rbd_data.115ee4742374.0000000000000018:head v 49'81, mlcod=49'79)", 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.070142910000001, 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045618899999999999, 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235801+0000", 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235887+0000", 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5688999999999999e-05 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235922+0000", 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5719000000000001e-05 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235941+0000", 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8916e-05 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.236065+0000", 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012414699999999999 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.236105+0000", 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0091e-05 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.240269+0000", 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041637339999999997 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.240361+0000", 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1310000000000005e-05 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.722860505, 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:34:56.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:34:56.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:34:56.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:34:56.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.72278665, 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.026091866000002, 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:34:56.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:22507 3.1f e63/59 3:ff4fbbf8:::rbd_data.115e6a380959.000000000000000c:head v 63'86, mlcod=63'84)", 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.545549742999999, 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052552429999999997, 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:34:56.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760396+0000", 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7290000000000002e-06 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760534+0000", 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138543 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760591+0000", 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7321000000000002e-05 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760612+0000", 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0239999999999999e-05 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760775+0000", 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016357000000000001 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760819+0000", 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3843e-05 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.765551+0000", 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047318270000000001 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.765647+0000", 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6169999999999996e-05 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33705 3.1f e70/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 70'124, mlcod=70'122)", 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8547237130000003, 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035074059999999998, 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451220+0000", 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0700000000000001e-06 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451266+0000", 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6059000000000001e-05 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451294+0000", 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8339000000000001e-05 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451311+0000", 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7076e-05 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451443+0000", 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013130999999999999 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451486+0000", 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3015000000000001e-05 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.454629+0000", 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003143046 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.454725+0000", 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6490999999999995e-05 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33828 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'128, mlcod=71'126)", 2024-04-24T00:34:56.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3621066919999993, 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00348315, 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:34:56.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943837+0000", 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6230000000000001e-06 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943901+0000", 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3255000000000003e-05 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943936+0000", 2024-04-24T00:34:56.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5191999999999998e-05 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943952+0000", 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6439999999999998e-05 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.944112+0000", 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000159946 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.944156+0000", 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3455000000000003e-05 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.947168+0000", 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030124259999999999 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.947318+0000", 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149813 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33843 3.0 e71/21 3:04e211c8:::rbd_data.115ec4f9b3a7.000000000000000f:head v 71'81, mlcod=69'79)", 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3262981019999991, 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041071550000000004, 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:34:56.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979645+0000", 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.302e-06 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979667+0000", 2024-04-24T00:34:56.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1821e-05 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979684+0000", 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7856999999999999e-05 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979694+0000", 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0049e-05 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979827+0000", 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013208800000000001 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979868+0000", 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1066000000000003e-05 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.983684+0000", 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038166749999999998 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.983751+0000", 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6297000000000006e-05 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2205599070000002, 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.372 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33935 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'135, mlcod=71'133)", 2024-04-24T00:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0841692960000007, 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052923950000000001, 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221773+0000", 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0109999999999999e-06 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221814+0000", 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0547999999999999e-05 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221829+0000", 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221839+0000", 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9399999999999997e-06 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221980+0000", 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014113999999999999 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.222046+0000", 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6107999999999998e-05 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.226980+0000", 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049344890000000002 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.227065+0000", 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4258000000000005e-05 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:35459 3.4 e74/21 3:26775e64:::rbd_object_map.115ec5833ea9:head v 74'96, mlcod=74'92)", 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.9210325600000004, 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043560150000000004, 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384910+0000", 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384917+0000", 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3599999999999998e-06 2024-04-24T00:34:56.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384933+0000", 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5821e-05 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384942+0000", 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8769999999999999e-06 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385004+0000", 2024-04-24T00:34:56.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2359000000000004e-05 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385027+0000", 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2878999999999999e-05 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389218+0000", 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004190399 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389265+0000", 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7445999999999999e-05 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.1f epoch 74/59 rep_tid 451 entries 74'138 (0'0) error 3:f8463f1b:::rbd_data.115ec5833ea9.0000000000000056:head by client.4446.0:35462 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 73'136)", 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.9208013069999996, 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041792909999999999, 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:56.385 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385141+0000", 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1600000000000002e-07 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385147+0000", 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9419999999999998e-06 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385170+0000", 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2966999999999999e-05 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385186+0000", 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5424999999999999e-05 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385266+0000", 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0408999999999995e-05 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389320+0000", 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040532320000000004 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.0 epoch 74/21 rep_tid 368 entries 74'86 (0'0) error 3:0193a4e3:::rbd_data.115ec5833ea9.0000000000000057:head by client.4446.0:35463 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 74'84)", 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.9201877849999995, 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035511779999999999, 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:34:56.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385755+0000", 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4099999999999997e-07 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385766+0000", 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1070000000000001e-05 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385786+0000", 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0492e-05 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385798+0000", 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1714e-05 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385883+0000", 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5068000000000006e-05 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389305+0000", 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034219929999999999 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:35490 3.9 e74/21 3:93eb3546:::rbd_data.115ec5833ea9.0000000000000020:head v 74'198, mlcod=74'196)", 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:34:56.392 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.8762448339999995, 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038708850000000001, 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429698+0000", 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0550000000000001e-06 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429742+0000", 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4011999999999998e-05 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429755+0000", 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3664e-05 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429762+0000", 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2210000000000002e-06 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429827+0000", 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5584000000000001e-05 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429853+0000", 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5817e-05 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.433515+0000", 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036616980000000001 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.433568+0000", 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2834000000000003e-05 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.397 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:56.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:34:56.399 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:56.395+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:56.500 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:56.494+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:56.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:34:56.633 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:56.630+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:56.702 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:56.695+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:56.713 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:34:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:34:56.713 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:56.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:34:56.803 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:56.799+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:56.904 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:56.898+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2036 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2031616~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.388035518999999, 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020639231000000001, 2024-04-24T00:34:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2036 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528463+0000", 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528464+0000", 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3600000000000004e-07 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528467+0000", 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8040000000000002e-06 2024-04-24T00:34:56.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528474+0000", 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7270000000000007e-06 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528480+0000", 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3340000000000001e-06 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548902+0000", 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020421963000000001 2024-04-24T00:34:56.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.548991+0000", 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9332000000000005e-05 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549102+0000", 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111535 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.924 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2053 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2097152~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387775347999998, 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616257999999998, 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2053 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528723+0000", 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528724+0000", 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6400000000000002e-07 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528727+0000", 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6749999999999998e-06 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528734+0000", 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4739999999999999e-06 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528740+0000", 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4759999999999997e-06 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549138+0000", 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020398315 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549225+0000", 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6802e-05 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549340+0000", 2024-04-24T00:34:56.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011495200000000001 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2054 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2228224~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387751481999999, 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020836107, 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2054 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528747+0000", 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528748+0000", 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999999e-07 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528750+0000", 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528756+0000", 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1249999999999998e-06 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528761+0000", 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549377+0000", 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020616254000000001 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549468+0000", 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1199000000000006e-05 2024-04-24T00:34:56.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.549583+0000", 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011484099999999999 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2055 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387730501999997, 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022112625, 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2055 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528769+0000", 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9299999999999999e-07 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528771+0000", 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2809999999999998e-06 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528777+0000", 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.127e-06 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528782+0000", 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8330000000000001e-06 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550648+0000", 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021866450999999999 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550755+0000", 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010676800000000001 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550881+0000", 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012577199999999999 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2056 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:56.937 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387708783999997, 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022390911999999999, 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2056 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528792+0000", 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1390000000000002e-06 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528798+0000", 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9869999999999996e-06 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528804+0000", 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4949999999999999e-06 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550928+0000", 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022124061 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551050+0000", 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121981 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551181+0000", 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130858 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387687839000002, 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.942 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387663924999998, 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.946 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:56.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387636751000002, 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.954 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387610645999999, 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.955 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387199664999997, 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:34:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387179856000003, 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:34:56.963 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.967 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387126797000001, 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:34:56.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.971 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387091941000001, 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387070377000001, 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387040884999998, 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:34:56.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.387020022999998, 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:34:56.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.386997448999999, 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:34:56.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.704737950999998, 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:34:56.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:56.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:34:56.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:56.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.704387036, 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:34:56.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:34:57.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.697498715999998, 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:57.009 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:57.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:34:57.010 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:57.006+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:57.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:34:57.114 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:57.110+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:57.132 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:34:57.133 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:34:57.133 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:34:57.133 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:34:57.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:34:57.208 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:34:57.212 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:57.207+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:57.243 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:57.243 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:57.260 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:57.260 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:57.279 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:57.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:57.280 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:57.280 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:57.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:34:57.299 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:57.299 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:57.319 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:57.319 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:57.338 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:34:57.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:34:57.342 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:57.339+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:57.414 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:57.407+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:57.461 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:57.461 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:57.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:34:57.519 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:57.515+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:57.615 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:57.610+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:57.641 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:57.642 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:57.642 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:34:57.642 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:34:57.642 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-24T00:34:57.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T00:34:57.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:34:57.716 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:57.711+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:57.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:34:57.776 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:57.776 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:57.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:34:57.834 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:57.831+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:57.901 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-04-24T00:34:57.917 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:57.911+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:57.959 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:57.959 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:57.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:34:57.974 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:34:57.974 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:57.974 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 80, 2024-04-24T00:34:57.974 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:34:57.974 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-04-24T00:33:24.857711+0000", 2024-04-24T00:34:57.974 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-04-24T00:34:57.699406+0000", 2024-04-24T00:34:57.974 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-04-24T00:34:51.642395+0000", 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-04-24T00:34:33.763713+0000", 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5806080, 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 4, 2024-04-24T00:34:57.975 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 3, 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:39.734814+0000", 2024-04-24T00:34:57.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-04-24T00:34:57.977 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:57.978 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "17", 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:57.979 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:57.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-04-24T00:34:57.981 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 8, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 8, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.25, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:34:57.982 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:43.786311+0000", 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:57.983 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:57.984 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "20", 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:57.985 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:57.986 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:57.987 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:57.988 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.9900000095367432, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.9900000095367432, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.630000114440918, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.630000114440918, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "pool_client.0", 2024-04-24T00:34:57.989 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:51.288702+0000", 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-04-24T00:34:57.990 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "80", 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:34:57.991 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 28, 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 80, 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:34:57.992 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:34:57.993 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:34:57.994 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.2400000095367432, 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.2400000095367432, 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1c80b3f4-99ee-4725-90a8-920116161bf5", 2024-04-24T00:34:57.995 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 0, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 53, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 75, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:57.996 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6817", 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6818", 2024-04-24T00:34:57.997 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6819", 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6822", 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6823", 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6820", 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6821", 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3307242121 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6817/3307242121", 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6819/3307242121", 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6823/3307242121", 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6821/3307242121", 2024-04-24T00:34:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "exists" 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "20e1331a-f8a2-48fd-b2b8-b7a92f21d78e", 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 59, 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6824", 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6825", 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6826", 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6827", 2024-04-24T00:34:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6830", 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6831", 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6828", 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6829", 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:34:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6825/637720411", 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6827/637720411", 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6831/637720411", 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6829/637720411", 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a51557a0-9634-4a20-82b0-b54cdd36b166", 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 55, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/820024667", 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6811/820024667", 2024-04-24T00:34:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6815/820024667", 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6813/820024667", 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "aafffa79-f532-4441-b1d6-f6441c5eebf8", 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.3392791748046875, 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 55, 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-04-24T00:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-04-24T00:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-04-24T00:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/1974534323", 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6803/1974534323", 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1974534323", 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1974534323", 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "78f05556-598e-4be2-b682-e3f58863eb82", 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 59, 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6816", 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6818", 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6820", 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6821", 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6826", 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6828", 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6823", 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6824", 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6818/1642543604", 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6821/1642543604", 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6828/1642543604", 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6824/1642543604", 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5", 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 70, 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6808", 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6809", 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6810", 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6811", 2024-04-24T00:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6814", 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6815", 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6812", 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6813", 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:34:58.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6809/172726194", 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6811/172726194", 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6815/172726194", 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6813/172726194", 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:34:58.029 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "6dad12bf-94de-482a-a402-49e67d5e5dc7", 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 59, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6800", 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6801", 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6802", 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6803", 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6806", 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6807", 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:58.033 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6804", 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6805", 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6801/2195746987", 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6803/2195746987", 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2195746987", 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6805/2195746987", 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e6734cfe-3c24-4ddd-8ef0-5ed450efe772", 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:34:58.035 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 62, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6817", 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:58.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6819", 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6822", 2024-04-24T00:34:58.037 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6825", 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.038 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6830", 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6831", 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6827", 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6829", 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6819/954791674", 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6825/954791674", 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6831/954791674", 2024-04-24T00:34:58.040 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6829/954791674", 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:58.041 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.509233+0000", 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 75 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.465696+0000", 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:58.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.405177+0000", 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:58.043 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.269202+0000", 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:58.044 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.359180+0000", 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.351834+0000", 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.045 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.490127+0000", 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:34:58.046 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.484842+0000", 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.4", 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.1b", 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-04-24T00:34:58.048 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [ 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:34:58.049 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 22, 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 24, 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 26, 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:58.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 28, 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [ 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 28, 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:34:58.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:58.052 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:58.055 INFO:tasks.thrashosds.thrasher:j is {'epoch': 80, 'fsid': '6a70d87d-4453-47be-af2a-5590f9851254', 'created': '2024-04-24T00:33:24.857711+0000', 'modified': '2024-04-24T00:34:57.699406+0000', 'last_up_change': '2024-04-24T00:34:51.642395+0000', 'last_in_change': '2024-04-24T00:34:33.763713+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T00:33:39.734814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T00:33:43.786311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-24T00:33:51.288702+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 28, 'snap_epoch': 80, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '1c80b3f4-99ee-4725-90a8-920116161bf5', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 53, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 3307242121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 3307242121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 3307242121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 3307242121}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 3307242121}]}, 'public_addr': '172.21.15.64:6817/3307242121', 'cluster_addr': '172.21.15.64:6819/3307242121', 'heartbeat_back_addr': '172.21.15.64:6823/3307242121', 'heartbeat_front_addr': '172.21.15.64:6821/3307242121', 'state': ['exists']}, {'osd': 1, 'uuid': '20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 59, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 637720411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 637720411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 637720411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 637720411}]}, 'public_addr': '172.21.15.64:6825/637720411', 'cluster_addr': '172.21.15.64:6827/637720411', 'heartbeat_back_addr': '172.21.15.64:6831/637720411', 'heartbeat_front_addr': '172.21.15.64:6829/637720411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a51557a0-9634-4a20-82b0-b54cdd36b166', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 820024667}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 820024667}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 820024667}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 820024667}]}, 'public_addr': '172.21.15.64:6809/820024667', 'cluster_addr': '172.21.15.64:6811/820024667', 'heartbeat_back_addr': '172.21.15.64:6815/820024667', 'heartbeat_front_addr': '172.21.15.64:6813/820024667', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aafffa79-f532-4441-b1d6-f6441c5eebf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3392791748046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1974534323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1974534323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1974534323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1974534323}]}, 'public_addr': '172.21.15.64:6801/1974534323', 'cluster_addr': '172.21.15.64:6803/1974534323', 'heartbeat_back_addr': '172.21.15.64:6807/1974534323', 'heartbeat_front_addr': '172.21.15.64:6805/1974534323', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '78f05556-598e-4be2-b682-e3f58863eb82', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 59, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6816', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6818', 'nonce': 1642543604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6820', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6821', 'nonce': 1642543604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6826', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6828', 'nonce': 1642543604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6823', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6824', 'nonce': 1642543604}]}, 'public_addr': '172.21.15.167:6818/1642543604', 'cluster_addr': '172.21.15.167:6821/1642543604', 'heartbeat_back_addr': '172.21.15.167:6828/1642543604', 'heartbeat_front_addr': '172.21.15.167:6824/1642543604', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6808', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 172726194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6810', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 172726194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6814', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 172726194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6812', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 172726194}]}, 'public_addr': '172.21.15.167:6809/172726194', 'cluster_addr': '172.21.15.167:6811/172726194', 'heartbeat_back_addr': '172.21.15.167:6815/172726194', 'heartbeat_front_addr': '172.21.15.167:6813/172726194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6dad12bf-94de-482a-a402-49e67d5e5dc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 59, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6800', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2195746987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6802', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2195746987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6806', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2195746987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6804', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2195746987}]}, 'public_addr': '172.21.15.167:6801/2195746987', 'cluster_addr': '172.21.15.167:6803/2195746987', 'heartbeat_back_addr': '172.21.15.167:6807/2195746987', 'heartbeat_front_addr': '172.21.15.167:6805/2195746987', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e6734cfe-3c24-4ddd-8ef0-5ed450efe772', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6817', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6819', 'nonce': 954791674}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6822', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6825', 'nonce': 954791674}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6830', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6831', 'nonce': 954791674}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6827', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6829', 'nonce': 954791674}]}, 'public_addr': '172.21.15.167:6819/954791674', 'cluster_addr': '172.21.15.167:6825/954791674', 'heartbeat_back_addr': '172.21.15.167:6831/954791674', 'heartbeat_front_addr': '172.21.15.167:6829/954791674', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.509233+0000', 'dead_epoch': 75}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.465696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.405177+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.269202+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.359180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.351834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.490127+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.484842+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.4', 'osds': [5, 7]}, {'pgid': '3.1b', 'osds': [5, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 22, 'length': 1}, {'begin': 24, 'length': 1}, {'begin': 26, 'length': 1}, {'begin': 28, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 28, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T00:34:58.055 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-24T00:34:58.055 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:34:58.047+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:34:58.155 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:58.150+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:58.179 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:58.179 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:58.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:34:58.255 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:58.250+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:58.295 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.295 INFO:teuthology.orchestra.run.smithi064.stdout:2039 read 0xbc6be2d thru 0xbc76411 (0xa5e5 bytes) 2024-04-24T00:34:58.296 INFO:teuthology.orchestra.run.smithi064.stdout:2040 write 0x6e1a89e thru 0x6e29794 (0xeef7 bytes) 2024-04-24T00:34:58.313 INFO:teuthology.orchestra.run.smithi064.stdout:2041 read 0xa62cab7 thru 0xa63b36c (0xe8b6 bytes) 2024-04-24T00:34:58.314 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.314 INFO:teuthology.orchestra.run.smithi064.stdout:2044 read 0x5ec9c6c thru 0x5ecd89f (0x3c34 bytes) 2024-04-24T00:34:58.315 INFO:teuthology.orchestra.run.smithi064.stdout:2045 read 0xa5168f2 thru 0xa526473 (0xfb82 bytes) 2024-04-24T00:34:58.320 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:58.320 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:58.330 INFO:teuthology.orchestra.run.smithi064.stdout:2046 writesame 0xd08f89c thru 0xd09b7a5 data_size 0xc6(0xbf0a bytes) 2024-04-24T00:34:58.333 INFO:teuthology.orchestra.run.smithi064.stdout:2047 write 0xc1b9230 thru 0xc1bab11 (0x18e2 bytes) 2024-04-24T00:34:58.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi064.stdout:2048 read 0x8847956 thru 0x885720d (0xf8b8 bytes) 2024-04-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi064.stdout:2049 punch from 0x8eeea7a to 0x8efd5a6, (0xeb2c bytes) 2024-04-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi064.stdout:2050 write 0x132faf5 thru 0x133d6df (0xdbeb bytes) 2024-04-24T00:34:58.347 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.356 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:58.350+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:58.359 INFO:teuthology.orchestra.run.smithi064.stdout:2052 writesame 0xed4b8e3 thru 0xed51f6a data_size 0x22(0x6688 bytes) 2024-04-24T00:34:58.362 INFO:teuthology.orchestra.run.smithi064.stdout:2053 punch from 0x7046bf6 to 0x70535ac, (0xc9b6 bytes) 2024-04-24T00:34:58.362 INFO:teuthology.orchestra.run.smithi064.stdout:2054 read 0x2ac4387 thru 0x2acb4f6 (0x7170 bytes) 2024-04-24T00:34:58.362 INFO:teuthology.orchestra.run.smithi064.stdout:2056 writesame 0x7386cd8 thru 0x7390a25 data_size 0x5(0x9d4e bytes) 2024-04-24T00:34:58.400 INFO:teuthology.orchestra.run.smithi064.stdout:2057 read 0xc3da59b thru 0xc3dd4cc (0x2f32 bytes) 2024-04-24T00:34:58.401 INFO:teuthology.orchestra.run.smithi064.stdout:2058 write 0x22bb793 thru 0x22c5471 (0x9cdf bytes) 2024-04-24T00:34:58.406 INFO:teuthology.orchestra.run.smithi064.stdout:2059 writesame 0xe035961 thru 0xe041ec2 data_size 0x5(0xc562 bytes) 2024-04-24T00:34:58.424 INFO:teuthology.orchestra.run.smithi064.stdout:2060 read 0xe8c2aee thru 0xe8c9bcc (0x70df bytes) 2024-04-24T00:34:58.457 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:58.451+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:58.463 INFO:teuthology.orchestra.run.smithi064.stdout:2061 trunc from 0xed51f6b to 0x9f9ca6 2024-04-24T00:34:58.464 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:58.464 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:58.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:34:58.497 INFO:teuthology.orchestra.run.smithi064.stdout:2062 read 0x64315 thru 0x71e9a (0xdb86 bytes) 2024-04-24T00:34:58.498 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.500 INFO:teuthology.orchestra.run.smithi064.stdout:2065 trunc from 0x9f9ca6 to 0x15377ab 2024-04-24T00:34:58.510 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.510 INFO:teuthology.orchestra.run.smithi064.stdout:2068 read 0x157a2b thru 0x165995 (0xdf6b bytes) 2024-04-24T00:34:58.511 INFO:teuthology.orchestra.run.smithi064.stdout:2070 punch from 0xdd04a0 to 0xdd7e77, (0x79d7 bytes) 2024-04-24T00:34:58.536 INFO:teuthology.orchestra.run.smithi064.stdout:2071 write 0x6513c9b thru 0x6520ee6 (0xd24c bytes) 2024-04-24T00:34:58.543 INFO:teuthology.orchestra.run.smithi064.stdout:2072 read 0x1131ccb thru 0x113b5bf (0x98f5 bytes) 2024-04-24T00:34:58.550 INFO:teuthology.orchestra.run.smithi064.stdout:2073 trunc from 0x6520ee7 to 0x411bac9 2024-04-24T00:34:58.558 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:34:58.554+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:34:58.616 INFO:teuthology.orchestra.run.smithi064.stdout:2074 write 0xe83a72c thru 0xe84574a (0xb01f bytes) 2024-04-24T00:34:58.627 INFO:teuthology.orchestra.run.smithi064.stdout:2076 punch from 0x1693287 to 0x169dc3a, (0xa9b3 bytes) 2024-04-24T00:34:58.627 INFO:teuthology.orchestra.run.smithi064.stdout:2077 read 0xab46c88 thru 0xab51151 (0xa4ca bytes) 2024-04-24T00:34:58.627 INFO:teuthology.orchestra.run.smithi064.stdout:2078 write 0x8ec256e thru 0x8ec4475 (0x1f08 bytes) 2024-04-24T00:34:58.635 INFO:teuthology.orchestra.run.smithi064.stdout:2079 read 0x8e42945 thru 0x8e47677 (0x4d33 bytes) 2024-04-24T00:34:58.635 INFO:teuthology.orchestra.run.smithi064.stdout:2081 writesame 0x74e15b9 thru 0x74eab8f data_size 0x1(0x95d7 bytes) 2024-04-24T00:34:58.656 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:58.656 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:58.660 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:58.655+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:58.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:34:58.729 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:34:58.754 INFO:teuthology.orchestra.run.smithi064.stdout:2082 read 0xe3b565e thru 0xe3c4b29 (0xf4cc bytes) 2024-04-24T00:34:58.756 INFO:teuthology.orchestra.run.smithi064.stdout:2083 writesame 0xb28f46d thru 0xb291ef0 data_size 0x6(0x2a84 bytes) 2024-04-24T00:34:58.758 INFO:teuthology.orchestra.run.smithi064.stdout:2084 read 0x17c5af4 thru 0x17d159f (0xbaac bytes) 2024-04-24T00:34:58.759 INFO:teuthology.orchestra.run.smithi064.stdout:2086 writesame 0xd7eee91 thru 0xd7fa4d1 data_size 0x25(0xb641 bytes) 2024-04-24T00:34:58.760 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:58.754+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:58.763 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.763 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.763 INFO:teuthology.orchestra.run.smithi064.stdout:2090 read 0xc589f6f thru 0xc596354 (0xc3e6 bytes) 2024-04-24T00:34:58.763 INFO:teuthology.orchestra.run.smithi064.stdout:2091 read 0xb792110 thru 0xb797a1b (0x590c bytes) 2024-04-24T00:34:58.765 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.765 INFO:teuthology.orchestra.run.smithi064.stdout:2093 write 0x22ad6d5 thru 0x22b1f15 (0x4841 bytes) 2024-04-24T00:34:58.768 INFO:teuthology.orchestra.run.smithi064.stdout:2095 writesame 0x1f41a24 thru 0x1f4e21b data_size 0x24(0xc7f8 bytes) 2024-04-24T00:34:58.773 INFO:teuthology.orchestra.run.smithi064.stdout:2096 write 0xcdfbec9 thru 0xce0539c (0x94d4 bytes) 2024-04-24T00:34:58.776 INFO:teuthology.orchestra.run.smithi064.stdout:2097 write 0x8865892 thru 0x886c2f4 (0x6a63 bytes) 2024-04-24T00:34:58.781 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.781 INFO:teuthology.orchestra.run.smithi064.stdout:2100 read 0x4729b8e thru 0x473348c (0x98ff bytes) 2024-04-24T00:34:58.781 INFO:teuthology.orchestra.run.smithi064.stdout:2102 read 0x2c41e50 thru 0x2c4aad4 (0x8c85 bytes) 2024-04-24T00:34:58.798 INFO:teuthology.orchestra.run.smithi064.stdout:2104 trunc from 0xe84574b to 0x88e1f9e 2024-04-24T00:34:58.818 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:34:58.832 INFO:teuthology.orchestra.run.smithi064.stdout:2107 write 0x9e08608 thru 0x9e0e1b9 (0x5bb2 bytes) 2024-04-24T00:34:58.838 INFO:teuthology.orchestra.run.smithi064.stdout:2108 punch from 0x6aba755 to 0x6abeb44, (0x43ef bytes) 2024-04-24T00:34:58.838 INFO:teuthology.orchestra.run.smithi064.stdout:2109 clone 14 order 18 su 8192 sc 2 2024-04-24T00:34:58.848 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:58.848 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:58.861 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:58.855+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:58.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:34:58.962 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:58.958+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:59.035 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:59.035 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:59.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:34:59.094 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:34:59.090+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:34:59.165 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:59.158+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:59.185 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:59.185 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:59.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:34:59.265 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:59.259+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:59.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:34:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:34:59.364 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:34:59.366 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:34:59.363+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:34:59.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:34:59.467 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:34:59.463+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:34:59.469 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:34:59.488 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:34:59.488 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:34:59.507 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:34:59.507 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:34:59.524 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:34:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:34:59.549 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:34:59.549 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2084 3.19 3:996477f0:::rbd_data.115eb9c76af6.0000000000000002:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.028022653999997, 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907047000000001, 2024-04-24T00:34:59.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2084 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530914+0000", 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3099999999999998e-07 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530922+0000", 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.498e-06 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530932+0000", 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0176e-05 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.530940+0000", 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4769999999999997e-06 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544421+0000", 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013481739 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544673+0000", 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025143600000000001 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.544821+0000", 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014819 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.124945289000003, 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:34:59.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:34:59.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.069433803000003, 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:34:59.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33687 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.175954108999999, 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015720043, 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33687 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382982+0000", 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382984+0000", 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.382994+0000", 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.008e-05 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383020+0000", 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5565999999999999e-05 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383034+0000", 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383167+0000", 2024-04-24T00:34:59.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132719 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.383325+0000", 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015864899999999999 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.384182+0000", 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085650199999999998 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398477+0000", 2024-04-24T00:34:59.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014294836 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398509+0000", 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2731999999999998e-05 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398615+0000", 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106084 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.398702+0000", 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7068000000000001e-05 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.968857155, 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:59.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:34:59.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.927292201, 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44584 3.6 3:616bc195:::rbd_data.115e56ad4d37.00000000000000c5:1b [read 305152~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860230+0000", 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6987063139999998, 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013907711999999999, 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.592 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44584 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860230+0000", 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860230+0000", 2024-04-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860231+0000", 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6300000000000005e-07 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860234+0000", 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1020000000000001e-06 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860243+0000", 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9949999999999994e-06 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860251+0000", 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7200000000000006e-06 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.873931+0000", 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013680269 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.874023+0000", 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1961999999999998e-05 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.874138+0000", 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011500100000000001 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44585 3.6 3:616bc195:::rbd_data.115e56ad4d37.00000000000000c5:1b [read 413696~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860261+0000", 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6986759999999999, 2024-04-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014348390000000001, 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44585 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860261+0000", 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860261+0000", 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860261+0000", 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1900000000000002e-07 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860264+0000", 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1310000000000002e-06 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860274+0000", 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4399999999999994e-06 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860281+0000", 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2250000000000003e-06 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.874205+0000", 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013924261 2024-04-24T00:34:59.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.874457+0000", 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164600000000001 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.874609+0000", 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015206799999999999 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.600 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44610 3.6 3:631fa8d6:::rbd_data.115e56ad4d37.00000000000000c1:1b [read 22528~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860316+0000", 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6986206099999999, 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014729463999999999, 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44610 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860316+0000", 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860316+0000", 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860316+0000", 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4099999999999999e-07 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860319+0000", 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.762e-06 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860331+0000", 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1474000000000001e-05 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860338+0000", 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4279999999999999e-06 2024-04-24T00:34:59.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.874664+0000", 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014326088000000001 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.874811+0000", 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000146386 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875045+0000", 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000234785 2024-04-24T00:34:59.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44611 3.6 3:631fa8d6:::rbd_data.115e56ad4d37.00000000000000c1:1b [read 131072~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860346+0000", 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6985901299999999, 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014933030999999999, 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44611 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860346+0000", 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860346+0000", 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860347+0000", 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3099999999999998e-07 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860349+0000", 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1689999999999999e-06 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860354+0000", 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2190000000000003e-06 2024-04-24T00:34:59.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860359+0000", 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.271e-06 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875100+0000", 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014740542000000001 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875178+0000", 2024-04-24T00:34:59.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7571999999999995e-05 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875279+0000", 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101827 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44612 3.6 3:631fa8d6:::rbd_data.115e56ad4d37.00000000000000c1:1b [read 241664~20480 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860366+0000", 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6985706189999998, 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015080437, 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44612 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860366+0000", 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860366+0000", 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860366+0000", 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3499999999999996e-07 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860369+0000", 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.807e-06 2024-04-24T00:34:59.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860378+0000", 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3280000000000006e-06 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860385+0000", 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3749999999999997e-06 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875310+0000", 2024-04-24T00:34:59.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014924873999999999 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875381+0000", 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1290000000000004e-05 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875446+0000", 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5228000000000006e-05 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44613 3.6 3:631fa8d6:::rbd_data.115e56ad4d37.00000000000000c1:1b [read 350208~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860394+0000", 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6985420549999999, 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015342817999999999, 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44613 2024-04-24T00:34:59.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860394+0000", 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860394+0000", 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860395+0000", 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3200000000000005e-07 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860406+0000", 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1284000000000001e-05 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860419+0000", 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2379e-05 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860426+0000", 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4100000000000002e-06 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875501+0000", 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01507476 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875600+0000", 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9325000000000002e-05 2024-04-24T00:34:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875737+0000", 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013712800000000001 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44614 3.6 3:631fa8d6:::rbd_data.115e56ad4d37.00000000000000c1:1b [read 458752~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860436+0000", 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.69850083, 2024-04-24T00:34:59.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015479533, 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44614 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860436+0000", 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860436+0000", 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860436+0000", 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1699999999999997e-07 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860439+0000", 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9059999999999998e-06 2024-04-24T00:34:59.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860462+0000", 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2844999999999999e-05 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860469+0000", 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9580000000000001e-06 2024-04-24T00:34:59.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875772+0000", 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015302969 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875832+0000", 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9936999999999999e-05 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875915+0000", 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3201000000000005e-05 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44630 3.6 3:616bc195:::rbd_data.115e56ad4d37.00000000000000c5:1b [read 20480~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860480+0000", 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.698456798, 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015733937, 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.623 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44630 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860480+0000", 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860480+0000", 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860480+0000", 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860483+0000", 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7970000000000001e-06 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860492+0000", 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5110000000000003e-06 2024-04-24T00:34:59.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860498+0000", 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8079999999999999e-06 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.875972+0000", 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015473351 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876063+0000", 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1130000000000003e-05 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876214+0000", 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015071 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44631 3.6 3:616bc195:::rbd_data.115e56ad4d37.00000000000000c5:1b [read 131072~20480 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860506+0000", 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.698430348, 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015890707, 2024-04-24T00:34:59.627 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44631 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860506+0000", 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860506+0000", 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860507+0000", 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1500000000000005e-07 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860509+0000", 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5909999999999998e-06 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860521+0000", 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1545e-05 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860528+0000", 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8650000000000003e-06 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876249+0000", 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015721776 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876329+0000", 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406000000000005e-05 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876397+0000", 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8009000000000006e-05 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44632 3.6 3:616bc195:::rbd_data.115e56ad4d37.00000000000000c5:1b [read 239616~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860536+0000", 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6984009740000001, 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016171802999999998, 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44632 2024-04-24T00:34:59.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860536+0000", 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860536+0000", 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860536+0000", 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7100000000000002e-07 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860539+0000", 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7980000000000001e-06 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860557+0000", 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8649999999999999e-05 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860565+0000", 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3710000000000002e-06 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876484+0000", 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015918772000000001 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876581+0000", 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7554999999999997e-05 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876707+0000", 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126186 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44633 3.6 3:616bc195:::rbd_data.115e56ad4d37.00000000000000c5:1b [read 348160~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860575+0000", 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.698361851, 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01629009, 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44633 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860575+0000", 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860575+0000", 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860575+0000", 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2500000000000005e-07 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860578+0000", 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.134e-06 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860588+0000", 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5829999999999996e-06 2024-04-24T00:34:59.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860595+0000", 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.221e-06 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876741+0000", 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016145309 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876786+0000", 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5621999999999998e-05 2024-04-24T00:34:59.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876865+0000", 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8596000000000004e-05 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44634 3.6 3:616bc195:::rbd_data.115e56ad4d37.00000000000000c5:1b [read 458752~20480 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.860603+0000", 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6983337650000001, 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016557751999999998, 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.640 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44634 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860603+0000", 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860603+0000", 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860603+0000", 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6899999999999998e-07 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860606+0000", 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6800000000000002e-06 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860614+0000", 2024-04-24T00:34:59.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2320000000000001e-06 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.860621+0000", 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8170000000000001e-06 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.876914+0000", 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016292613000000001 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.877034+0000", 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120124 2024-04-24T00:34:59.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.877160+0000", 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012681699999999999 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44651 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head [write 0~524288 [fadvise_sequential+fadvise_nocache],write 655360~131072 [fadvise_sequential+fadvise_nocache],write 1310720~1048576 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~4096 [fadvise_sequential+fadvise_nocache],write 11155456~4096 [fadvise_sequential+fadvise_nocache],write 11167744~4096 [fadvise_sequential+fadvise_nocache],write 11180032~4096 [fadvise_sequential+fadvise_nocache],write 11192320~4096 [fadvise_sequential+fadvise_nocache],write 11204608~4096 [fadvise_sequential+fadvise_nocache],write 11216896~4096 [fadvise_sequential+fadvise_nocache],write 11229184~4096 [fadvise_sequential+fadvise_nocache],write 11241472~4096 [fadvise_sequential+fadvise_nocache],write 11253760~4096 [fadvise_sequential+fadvise_nocache],write 11266048~4096 [fadvise_sequential+fadvise_nocache],write 14548992~8192 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~12288 [fadvise_sequential+fadvise_nocache],write 14688256~8192 [fadvise_sequential+fadvise_nocache],write 14700544~8192 [fadvise_sequential+fadvise_nocache],write 14712832~8192 [fadvise_sequential+fadvise_nocache],write 14725120~8192 [fadvise_sequential+fadvise_nocache],write 14737408~8192 [fadvise_sequential+fadvise_nocache],write 14749696~8192 [fadvise_sequential+fadvise_nocache],write 14761984~8192 [fadvise_sequential+fadvise_nocache],write 14774272~8192 [fadvise_sequential+fadvise_nocache],write 14786560~8192 [fadvise_sequential+fadvise_nocache],write 14798848~6453 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6746323360000002, 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058685117000000002, 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44651 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884307+0000", 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0850000000000001e-06 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886333+0000", 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020258310000000001 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886370+0000", 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6485000000000001e-05 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886385+0000", 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5897e-05 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886564+0000", 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178455 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.887161+0000", 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059714499999999999 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.888890+0000", 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001729349 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920023+0000", 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031132238999999999 2024-04-24T00:34:59.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942736+0000", 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02271306 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942884+0000", 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014812199999999999 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942989+0000", 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105449 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44651 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head [write 0~524288 [fadvise_sequential+fadvise_nocache],write 655360~131072 [fadvise_sequential+fadvise_nocache],write 1310720~1048576 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~4096 [fadvise_sequential+fadvise_nocache],write 11155456~4096 [fadvise_sequential+fadvise_nocache],write 11167744~4096 [fadvise_sequential+fadvise_nocache],write 11180032~4096 [fadvise_sequential+fadvise_nocache],write 11192320~4096 [fadvise_sequential+fadvise_nocache],write 11204608~4096 [fadvise_sequential+fadvise_nocache],write 11216896~4096 [fadvise_sequential+fadvise_nocache],write 11229184~4096 [fadvise_sequential+fadvise_nocache],write 11241472~4096 [fadvise_sequential+fadvise_nocache],write 11253760~4096 [fadvise_sequential+fadvise_nocache],write 11266048~4096 [fadvise_sequential+fadvise_nocache],write 14548992~8192 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~12288 [fadvise_sequential+fadvise_nocache],write 14688256~8192 [fadvise_sequential+fadvise_nocache],write 14700544~8192 [fadvise_sequential+fadvise_nocache],write 14712832~8192 [fadvise_sequential+fadvise_nocache],write 14725120~8192 [fadvise_sequential+fadvise_nocache],write 14737408~8192 [fadvise_sequential+fadvise_nocache],write 14749696~8192 [fadvise_sequential+fadvise_nocache],write 14761984~8192 [fadvise_sequential+fadvise_nocache],write 14774272~8192 [fadvise_sequential+fadvise_nocache],write 14786560~8192 [fadvise_sequential+fadvise_nocache],write 14798848~6453 [fadvise_sequential+fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.928302+0000", 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.6306349669999998, 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014622191, 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:34:59.650 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for ondisk", 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44651 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.928302+0000", 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:34:59.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.928302+0000", 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.928304+0000", 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.929456+0000", 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011519360000000001 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:34:59.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.929478+0000", 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2300999999999999e-05 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.929489+0000", 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0835999999999999e-05 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.929775+0000", 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000285851 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for ondisk", 2024-04-24T00:34:59.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.929991+0000", 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021643899999999999 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942924+0000", 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012932740999999999 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:34:59.654 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:59.655 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:34:59.655 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:34:59.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:34:59.657 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:34:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:34:59.657 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:34:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:34:59.658 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:34:59.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:34:59.659 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:34:59.651+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:34:59.757 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:59.750+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:59.835 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:34:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:34:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:34:59.836 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:34:59.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:34:59.894 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:59.890+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:34:59.958 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:34:59.954+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:00.025 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:00.025 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:00.025 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:00.025 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:00.025 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:00.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:00.082 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:00.078+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:00.161 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:00.155+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:2979 3.9 e33/21 3:9717bf35:::rbd_data.115e5ff16b0c.000000000000001b:head v 33'28, mlcod=30'26)", 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.379233474999999, 2024-04-24T00:35:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053184019999999999, 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822202+0000", 2024-04-24T00:35:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822203+0000", 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3909999999999999e-06 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822482+0000", 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027925399999999999 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822509+0000", 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.707e-05 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822521+0000", 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1779e-05 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822626+0000", 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105165 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.822653+0000", 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6713e-05 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827477+0000", 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048240949999999996 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.827520+0000", 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2935000000000001e-05 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:35:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.556628842000002, 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:35:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10425 3.4 e41/21 3:2657577c:::rbd_data.115e380d621c.0000000000000210:head v 41'28, mlcod=39'26)", 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.583158167999997, 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041008809999999998, 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618277+0000", 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618285+0000", 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5580000000000003e-06 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618309+0000", 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-05 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.618321+0000", 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.164e-05 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619328+0000", 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010071259999999999 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.619360+0000", 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1581000000000001e-05 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622325+0000", 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029648610000000001 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.622378+0000", 2024-04-24T00:35:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3235000000000001e-05 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:10705 3.9 e41/21 3:93f8bbf0:::rbd_data.115e380d621c.0000000000000399:head v 41'63, mlcod=41'61)", 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.039198143, 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034786589999999998, 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162237+0000", 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162238+0000", 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3699999999999999e-07 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162284+0000", 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6122999999999999e-05 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162296+0000", 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1871e-05 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162304+0000", 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7030000000000006e-06 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162390+0000", 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6582999999999994e-05 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.162431+0000", 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0577999999999997e-05 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165655+0000", 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003224612 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:14.165716+0000", 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0251999999999998e-05 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20211 3.4 e48/21 3:249c5675:::rbd_data.115ee4742374.0000000000000021:head v 48'33, mlcod=41'31)", 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.920438363999999, 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004558655, 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280997+0000", 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.280999+0000", 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0559999999999999e-06 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281071+0000", 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2300999999999998e-05 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281102+0000", 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1078000000000003e-05 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281116+0000", 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4273e-05 2024-04-24T00:35:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281198+0000", 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1364000000000002e-05 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.281221+0000", 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3589999999999999e-05 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.285420+0000", 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041987559999999997 2024-04-24T00:35:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.285555+0000", 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013523699999999999 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20552 3.9 e49/21 3:923662ec:::rbd_data.115ee4742374.000000000000000a:head v 49'88, mlcod=48'86)", 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.227935242000001, 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062734230000000002, 2024-04-24T00:35:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973502+0000", 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.04e-06 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973677+0000", 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473200000000001 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973714+0000", 2024-04-24T00:35:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7117999999999999e-05 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973734+0000", 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0102000000000002e-05 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973879+0000", 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145654 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973946+0000", 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6476999999999994e-05 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979687+0000", 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057411140000000003 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979773+0000", 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6186000000000005e-05 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20615 3.1f e49/21 3:fddc1920:::rbd_data.115ee4742374.0000000000000018:head v 49'81, mlcod=49'79)", 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.965636390999997, 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045618899999999999, 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:35:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235799+0000", 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235801+0000", 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235887+0000", 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5688999999999999e-05 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235922+0000", 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5719000000000001e-05 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.235941+0000", 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8916e-05 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.236065+0000", 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012414699999999999 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.236105+0000", 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0091e-05 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.240269+0000", 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041637339999999997 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:23.240361+0000", 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1310000000000005e-05 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.618353986000002, 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:35:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:35:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.618280130999999, 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.921585346999997, 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:35:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:22507 3.1f e63/59 3:ff4fbbf8:::rbd_data.115e6a380959.000000000000000c:head v 63'86, mlcod=63'84)", 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.441043224000001, 2024-04-24T00:35:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052552429999999997, 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760392+0000", 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760396+0000", 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7290000000000002e-06 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760534+0000", 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138543 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760591+0000", 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7321000000000002e-05 2024-04-24T00:35:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760612+0000", 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0239999999999999e-05 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760775+0000", 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016357000000000001 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.760819+0000", 2024-04-24T00:35:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3843e-05 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.765551+0000", 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047318270000000001 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:37.765647+0000", 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6169999999999996e-05 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33705 3.1f e70/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 70'124, mlcod=70'122)", 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.750217193999999, 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035074059999999998, 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451218+0000", 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451220+0000", 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0700000000000001e-06 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451266+0000", 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6059000000000001e-05 2024-04-24T00:35:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451294+0000", 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8339000000000001e-05 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451311+0000", 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7076e-05 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451443+0000", 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013130999999999999 2024-04-24T00:35:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.451486+0000", 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3015000000000001e-05 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.454629+0000", 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003143046 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.454725+0000", 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6490999999999995e-05 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33828 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'128, mlcod=71'126)", 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.257600173, 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00348315, 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:35:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943835+0000", 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943837+0000", 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6230000000000001e-06 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943901+0000", 2024-04-24T00:35:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3255000000000003e-05 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943936+0000", 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5191999999999998e-05 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.943952+0000", 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6439999999999998e-05 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.944112+0000", 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000159946 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.944156+0000", 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3455000000000003e-05 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.947168+0000", 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030124259999999999 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.947318+0000", 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149813 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33843 3.0 e71/21 3:04e211c8:::rbd_data.115ec4f9b3a7.000000000000000f:head v 71'81, mlcod=69'79)", 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.221791583, 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041071550000000004, 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979643+0000", 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979645+0000", 2024-04-24T00:35:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.302e-06 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979667+0000", 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1821e-05 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979684+0000", 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7856999999999999e-05 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979694+0000", 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0049e-05 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979827+0000", 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013208800000000001 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.979868+0000", 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1066000000000003e-05 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.983684+0000", 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038166749999999998 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.983751+0000", 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6297000000000006e-05 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.116053387999999, 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:35:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:35:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:35:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33935 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'135, mlcod=71'133)", 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.979662777, 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052923950000000001, 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221772+0000", 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221773+0000", 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0109999999999999e-06 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221814+0000", 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0547999999999999e-05 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221829+0000", 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221839+0000", 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9399999999999997e-06 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.221980+0000", 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014113999999999999 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.222046+0000", 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6107999999999998e-05 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.226980+0000", 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049344890000000002 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.227065+0000", 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4258000000000005e-05 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:35459 3.4 e74/21 3:26775e64:::rbd_object_map.115ec5833ea9:head v 74'96, mlcod=74'92)", 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:35:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8165260409999995, 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043560150000000004, 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384909+0000", 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384910+0000", 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384917+0000", 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3599999999999998e-06 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384933+0000", 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5821e-05 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.384942+0000", 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8769999999999999e-06 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385004+0000", 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2359000000000004e-05 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385027+0000", 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2878999999999999e-05 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389218+0000", 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004190399 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389265+0000", 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7445999999999999e-05 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.1f epoch 74/59 rep_tid 451 entries 74'138 (0'0) error 3:f8463f1b:::rbd_data.115ec5833ea9.0000000000000056:head by client.4446.0:35462 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 73'136)", 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8162947880000004, 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041792909999999999, 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385140+0000", 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385141+0000", 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1600000000000002e-07 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385147+0000", 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9419999999999998e-06 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385170+0000", 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2966999999999999e-05 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385186+0000", 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5424999999999999e-05 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385266+0000", 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0408999999999995e-05 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389320+0000", 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040532320000000004 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.0 epoch 74/21 rep_tid 368 entries 74'86 (0'0) error 3:0193a4e3:::rbd_data.115ec5833ea9.0000000000000057:head by client.4446.0:35463 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 74'84)", 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8156812660000003, 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035511779999999999, 2024-04-24T00:35:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385754+0000", 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385755+0000", 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4099999999999997e-07 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385766+0000", 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1070000000000001e-05 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385786+0000", 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0492e-05 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385798+0000", 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1714e-05 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.385883+0000", 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5068000000000006e-05 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.389305+0000", 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034219929999999999 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:35490 3.9 e74/21 3:93eb3546:::rbd_data.115ec5833ea9.0000000000000020:head v 74'198, mlcod=74'196)", 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.7717383150000003, 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038708850000000001, 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429697+0000", 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429698+0000", 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0550000000000001e-06 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429742+0000", 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4011999999999998e-05 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429755+0000", 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3664e-05 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429762+0000", 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2210000000000002e-06 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429827+0000", 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5584000000000001e-05 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.429853+0000", 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5817e-05 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.433515+0000", 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036616980000000001 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:50.433568+0000", 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2834000000000003e-05 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.293 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:00.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:00.294 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:00.287+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:00.395 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:00.390+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:00.424 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:00.424 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:00.424 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:00.425 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:00.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:00.496 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:00.491+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:00.597 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:00.590+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:00.614 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:00.614 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:00.615 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:00.615 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:00.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:00.717 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:00.714+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:00.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:35:00.799 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:00.795+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:00.845 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2055 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2359296~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312915082000004, 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022112625, 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2055 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528768+0000", 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528769+0000", 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9299999999999999e-07 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528771+0000", 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2809999999999998e-06 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528777+0000", 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.127e-06 2024-04-24T00:35:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528782+0000", 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8330000000000001e-06 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550648+0000", 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021866450999999999 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550755+0000", 2024-04-24T00:35:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010676800000000001 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550881+0000", 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012577199999999999 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2056 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2490368~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312893363999997, 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022390911999999999, 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2056 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528790+0000", 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528792+0000", 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1390000000000002e-06 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528798+0000", 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9869999999999996e-06 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528804+0000", 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4949999999999999e-06 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.550928+0000", 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022124061 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551050+0000", 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121981 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551181+0000", 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130858 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312872419000001, 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:35:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:35:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312848504999998, 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312821331000002, 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:35:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312795225999999, 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:35:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:35:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:35:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312384244999997, 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312364436000003, 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312311377, 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312276521000001, 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312254957, 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:35:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:35:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312225464999997, 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312204602999998, 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:35:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:35:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.312182028999999, 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.629922530999998, 2024-04-24T00:35:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:35:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.629571616, 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:35:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:35:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.622683295999998, 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:35:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44653 3.c e79/56 3:3545ecd3:::rbd_data.115ec7ed39de.0000000000000002:head v 79'268, mlcod=77'266)", 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9519395749999999, 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022668765, 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7800000000000002e-07 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892860+0000", 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003115821 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892903+0000", 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3176999999999997e-05 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892926+0000", 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2459999999999998e-05 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.893026+0000", 2024-04-24T00:35:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010062 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.893059+0000", 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2806999999999999e-05 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.912296+0000", 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019237431999999999 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.912412+0000", 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011597 2024-04-24T00:35:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44656 3.7 e79/44 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:head v 79'103, mlcod=77'101)", 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9473568299999999, 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026507441999999999, 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894327+0000", 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1e-07 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898194+0000", 2024-04-24T00:35:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038672889999999999 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898203+0000", 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6340000000000007e-06 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898208+0000", 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2009999999999998e-06 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898262+0000", 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4466999999999999e-05 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898280+0000", 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7492000000000001e-05 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920764+0000", 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022483974 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920834+0000", 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0174999999999998e-05 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44661 3.17 e79/59 3:eff1c653:::rbd_data.115ec7ed39de.0000000000000005:head v 79'91, mlcod=77'89)", 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9454895410000002, 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024065629000000002, 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8299999999999996e-07 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897771+0000", 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015761690000000001 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897786+0000", 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5051999999999999e-05 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897794+0000", 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4689999999999997e-06 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897869+0000", 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.47e-05 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897899+0000", 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9777e-05 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920181+0000", 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022282560999999999 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920259+0000", 2024-04-24T00:35:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8218000000000002e-05 2024-04-24T00:35:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:00.931 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:00.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:00.933 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:00.927+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:01.033 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:01.027+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:01.135 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:01.131+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:01.143 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:53158 3.9 3:9717874c:::rbd_data.115e8561432.00000000000000b0:head [write 0~262144 [fadvise_sequential+fadvise_nocache] in=262144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:01.138238+0000", 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0038808549999999999, 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003904037, 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 53158 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138238+0000", 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138238+0000", 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138239+0000", 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8250000000000001e-06 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138442+0000", 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202544 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138471+0000", 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9082e-05 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138481+0000", 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8160000000000005e-06 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138556+0000", 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5475000000000005e-05 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138712+0000", 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015612699999999999 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.138876+0000", 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000163602 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.141579+0000", 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027027560000000002 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:53160 3.a e83/21 3:55bda10f:::rbd_data.115e8561432.00000000000000b2:head v 83'239, mlcod=83'237)", 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:01.140644+0000", 2024-04-24T00:35:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00147401, 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015482219999999999, 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.140644+0000", 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.140644+0000", 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.140645+0000", 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9500000000000001e-07 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.140753+0000", 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010824599999999999 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.140765+0000", 2024-04-24T00:35:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1742e-05 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.140771+0000", 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3650000000000003e-06 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.140821+0000", 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0275000000000002e-05 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:01.140845+0000", 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3782999999999999e-05 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:35:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 2 2024-04-24T00:35:01.152 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:01.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:01.258 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:01.255+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:01.336 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:01.331+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:01.406 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:01.407 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:01.407 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:01.407 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:01.407 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:01.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:01.474 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:01.471+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:01.538 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:01.535+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:01.639 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:01.634+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.010330611999997, 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:35:01.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:35:01.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.010254412999998, 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.660 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:35:01.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:35:01.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:35:01.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.009796239000003, 2024-04-24T00:35:01.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.009738931999998, 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.669 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:35:01.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:35:01.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:35:01.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.009479966000001, 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.673 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.008558493999999, 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.677 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:01.681 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.436743763999999, 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:01.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:01.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:01.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:01.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:01.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:01.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:01.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:01.691 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.421214515000003, 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:01.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:01.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.792491194999997, 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:01.695 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:01.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:01.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:01.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.777456067999999, 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:35:01.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.014958069999999, 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:01.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:01.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:01.710 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.001135112, 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:01.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.773993926999999, 2024-04-24T00:35:01.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:01.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:01.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:01.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.762204276999999, 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.720 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30704 3.1a 3:5e74e97a:::rbd_data.115e8210e51a.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.398712358000001, 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043351496000000003, 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.724 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30704 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252358+0000", 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1790000000000001e-06 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252364+0000", 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6549999999999998e-06 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252381+0000", 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6767000000000001e-05 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252390+0000", 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9959999999999999e-06 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252492+0000", 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102563 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252668+0000", 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017553499999999999 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.254545+0000", 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001877037 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295422+0000", 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040876527000000003 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295513+0000", 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1838999999999999e-05 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295614+0000", 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010027900000000001 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295708+0000", 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4118999999999998e-05 2024-04-24T00:35:01.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.768164213, 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:01.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:01.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:01.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33518 3.b e70/59 ondisk, result = 0)", 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.756840685, 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042187448000000002, 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894230+0000", 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7680000000000001e-06 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894239+0000", 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3319999999999998e-06 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894266+0000", 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049000000000001e-05 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894281+0000", 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5245000000000001e-05 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936247+0000", 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041965833000000001 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936257+0000", 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936415+0000", 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158605 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.739 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33734 3.b 3:d4ebe596:::rbd_data.115ec4f9b3a7.000000000000000e:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.13179849, 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041668288999999997, 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33734 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519272+0000", 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9870000000000002e-06 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519282+0000", 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0203e-05 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519310+0000", 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7880000000000001e-05 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519324+0000", 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519402+0000", 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7865000000000001e-05 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519481+0000", 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9103000000000005e-05 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.522670+0000", 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003189 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560812+0000", 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038141427999999998 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560831+0000", 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9820999999999999e-05 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560886+0000", 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5087999999999998e-05 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560938+0000", 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1993999999999997e-05 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44651 3.6 e79/59 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head v 79'72, mlcod=77'70)", 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.7620911220000002, 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048315631999999997, 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:01.745 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892459+0000", 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034802259999999999 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892475+0000", 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5998000000000002e-05 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892485+0000", 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8679999999999994e-06 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892653+0000", 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016800800000000001 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892721+0000", 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8953000000000002e-05 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937243+0000", 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044521666000000001 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937293+0000", 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0043000000000002e-05 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44659 3.1d e79/53 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head v 79'105, mlcod=77'103)", 2024-04-24T00:35:01.749 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.7546099389999998, 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038358476000000002, 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:01.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896460+0000", 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2399999999999997e-07 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897721+0000", 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012617049999999999 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897744+0000", 2024-04-24T00:35:01.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2982000000000001e-05 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897756+0000", 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1559999999999999e-05 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897836+0000", 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0434999999999998e-05 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:01.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897871+0000", 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5147000000000001e-05 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934696+0000", 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036824511999999997 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934817+0000", 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121311 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.754 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:01.754 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:01.754 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:01.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:01.755 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:35:01.756 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:01.750+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:01.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:01.857 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:01.850+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:01.914 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:35:01.932 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:01.932 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:01.949 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:01.949 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:01.957 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:01.957 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:01.957 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:01.957 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:01.957 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:01.951+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:01.965 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:01.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:01.983 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:02.019 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:02.020 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:02.039 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:02.039 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:35:02.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:35:02.058 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:02.055+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:02.135 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:02.135 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:02.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:02.198 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:02.195+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:02.260 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:02.254+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:02.353 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:02.353 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:02.360 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:02.355+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:02.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:02.462 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:02.455+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:02.520 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:02.520 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:02.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:02.574 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:02.571+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:02.664 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:02.659+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:02.688 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:02.688 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:02.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:02.786 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:02.783+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:02.853 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:02.853 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:02.865 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:02.866 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:02.862+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:02.945 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-04-24T00:35:02.967 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:02.962+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:03.015 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:03.015 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:03.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:03.056 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:35:03.056 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:35:03.056 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-24T00:35:03.056 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:35:03.056 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:35:03.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-24T00:35:03.070 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:03.067+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:03.147 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:03.148 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:03.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:03.169 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:03.162+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:03.177 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:03.177 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T00:35:03.177 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:03.269 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:03.266+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:03.275 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:03.275 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:03.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:03.370 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:03.366+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:03.409 INFO:teuthology.orchestra.run.smithi064.stdout:2110 write 0xe0f48a2 thru 0xe10363e (0xed9d bytes) 2024-04-24T00:35:03.414 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:03.414 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:03.426 INFO:teuthology.orchestra.run.smithi064.stdout:2112 read 0xde7b241 thru 0xde7cd30 (0x1af0 bytes) 2024-04-24T00:35:03.426 INFO:teuthology.orchestra.run.smithi064.stdout:2113 punch from 0xe151ce to 0xe20b39, (0xb96b bytes) 2024-04-24T00:35:03.426 INFO:teuthology.orchestra.run.smithi064.stdout:2114 read 0x8189276 thru 0x8191a4c (0x87d7 bytes) 2024-04-24T00:35:03.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:03.430 INFO:teuthology.orchestra.run.smithi064.stdout:2116 read 0xd082266 thru 0xd0915c5 (0xf360 bytes) 2024-04-24T00:35:03.430 INFO:teuthology.orchestra.run.smithi064.stdout:2117 write 0x2783757 thru 0x2791d55 (0xe5ff bytes) 2024-04-24T00:35:03.435 INFO:teuthology.orchestra.run.smithi064.stdout:2118 write 0x1fa96ec thru 0x1fb1b38 (0x844d bytes) 2024-04-24T00:35:03.440 INFO:teuthology.orchestra.run.smithi064.stdout:2120 read 0xba37af9 thru 0xba3e2db (0x67e3 bytes) 2024-04-24T00:35:03.440 INFO:teuthology.orchestra.run.smithi064.stdout:2121 read 0x4b7468 thru 0x4bde04 (0x699d bytes) 2024-04-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:2123 trunc from 0xe10363f to 0x9c5b3fd 2024-04-24T00:35:03.471 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:03.467+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:03.474 INFO:teuthology.orchestra.run.smithi064.stdout:2124 write 0x8ab896 thru 0x8b9559 (0xdcc4 bytes) 2024-04-24T00:35:03.481 INFO:teuthology.orchestra.run.smithi064.stdout:2125 read 0x7d1f285 thru 0x7d29d83 (0xaaff bytes) 2024-04-24T00:35:03.484 INFO:teuthology.orchestra.run.smithi064.stdout:2126 write 0x1481227 thru 0x148cc41 (0xba1b bytes) 2024-04-24T00:35:03.509 INFO:teuthology.orchestra.run.smithi064.stdout:2127 write 0xe9cdc4b thru 0xe9d90bc (0xb472 bytes) 2024-04-24T00:35:03.520 INFO:teuthology.orchestra.run.smithi064.stdout:2128 read 0xccec7a2 thru 0xccf9bc9 (0xd428 bytes) 2024-04-24T00:35:03.520 INFO:teuthology.orchestra.run.smithi064.stdout:2130 write 0x3f81105 thru 0x3f88bae (0x7aaa bytes) 2024-04-24T00:35:03.525 INFO:teuthology.orchestra.run.smithi064.stdout:2134 punch from 0xb160ded to 0xb162948, (0x1b5b bytes) 2024-04-24T00:35:03.525 INFO:teuthology.orchestra.run.smithi064.stdout:2135 writesame 0x48c01a5 thru 0x48c76ef data_size 0x1(0x754b bytes) 2024-04-24T00:35:03.560 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:03.560 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:03.572 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:03.567+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:03.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:03.594 INFO:teuthology.orchestra.run.smithi064.stdout:2136 writesame 0x5a0fbe2 thru 0x5a1ea47 data_size 0x55(0xee66 bytes) 2024-04-24T00:35:03.598 INFO:teuthology.orchestra.run.smithi064.stdout:2137 writesame 0x56dbfa0 thru 0x56eb3a8 data_size 0x1(0xf409 bytes) 2024-04-24T00:35:03.673 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:03.667+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:03.718 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:03.718 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:03.724 INFO:teuthology.orchestra.run.smithi064.stdout:2139 read 0xa548ed3 thru 0xa554735 (0xb863 bytes) 2024-04-24T00:35:03.724 INFO:teuthology.orchestra.run.smithi064.stdout:2140 read 0x941e14b thru 0x942ce64 (0xed1a bytes) 2024-04-24T00:35:03.726 INFO:teuthology.orchestra.run.smithi064.stdout:2141 write 0x29d4c9d thru 0x29d62a3 (0x1607 bytes) 2024-04-24T00:35:03.732 INFO:teuthology.orchestra.run.smithi064.stdout:2143 punch from 0x4cd4da3 to 0x4ce1f75, (0xd1d2 bytes) 2024-04-24T00:35:03.732 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:03.732 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:03.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:03.747 INFO:teuthology.orchestra.run.smithi064.stdout:2148 trunc from 0xe9d90bd to 0x553fb95 2024-04-24T00:35:03.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:35:03.773 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:03.767+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:03.874 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:03.867+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:03.880 INFO:teuthology.orchestra.run.smithi064.stdout:2150 writesame 0xbfea048 thru 0xbfed125 data_size 0xa(0x30de bytes) 2024-04-24T00:35:03.880 INFO:teuthology.orchestra.run.smithi064.stdout:2151 write 0x460d97b thru 0x4618652 (0xacd8 bytes) 2024-04-24T00:35:03.893 INFO:teuthology.orchestra.run.smithi064.stdout:2152 write 0xcda319e thru 0xcdaa374 (0x71d7 bytes) 2024-04-24T00:35:03.897 INFO:teuthology.orchestra.run.smithi064.stdout:2153 write 0x8f79006 thru 0x8f7a0fd (0x10f8 bytes) 2024-04-24T00:35:03.898 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:03.898 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:03.902 INFO:teuthology.orchestra.run.smithi064.stdout:2154 write 0x2cd5116 thru 0x2ce4fb2 (0xfe9d bytes) 2024-04-24T00:35:03.906 INFO:teuthology.orchestra.run.smithi064.stdout:2155 read 0x1572da5 thru 0x157e798 (0xb9f4 bytes) 2024-04-24T00:35:03.907 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:03.907 INFO:teuthology.orchestra.run.smithi064.stdout:2160 read 0x9b3d4d9 thru 0x9b49bb4 (0xc6dc bytes) 2024-04-24T00:35:03.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:03.924 INFO:teuthology.orchestra.run.smithi064.stdout:2163 trunc from 0xcdaa375 to 0x28d1b62 2024-04-24T00:35:03.976 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:03.971+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:03.997 INFO:teuthology.orchestra.run.smithi064.stdout:2164 trunc from 0x28d1b62 to 0x76ad996 2024-04-24T00:35:04.005 INFO:teuthology.orchestra.run.smithi064.stdout:2165 read 0x10054bd thru 0x10056c2 (0x206 bytes) 2024-04-24T00:35:04.005 INFO:teuthology.orchestra.run.smithi064.stdout:2166 read 0x63c326 thru 0x64b9f8 (0xf6d3 bytes) 2024-04-24T00:35:04.018 INFO:teuthology.orchestra.run.smithi064.stdout:2167 write 0x9ab7a3c thru 0x9abe7f8 (0x6dbd bytes) 2024-04-24T00:35:04.029 INFO:teuthology.orchestra.run.smithi064.stdout:2169 write 0x7d322b5 thru 0x7d334a3 (0x11ef bytes) 2024-04-24T00:35:04.031 INFO:teuthology.orchestra.run.smithi064.stdout:2170 read 0x5cf29ac thru 0x5d00639 (0xdc8e bytes) 2024-04-24T00:35:04.031 INFO:teuthology.orchestra.run.smithi064.stdout:2171 read 0x8224728 thru 0x822e4d2 (0x9dab bytes) 2024-04-24T00:35:04.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:04.048 INFO:teuthology.orchestra.run.smithi064.stdout:2172 write 0xdc59614 thru 0xdc605d3 (0x6fc0 bytes) 2024-04-24T00:35:04.057 INFO:teuthology.orchestra.run.smithi064.stdout:2174 write 0xd84d6ee thru 0xd84f4e7 (0x1dfa bytes) 2024-04-24T00:35:04.062 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:04.062 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:04.066 INFO:teuthology.orchestra.run.smithi064.stdout:2176 read 0xbaaf265 thru 0xbaba4a3 (0xb23f bytes) 2024-04-24T00:35:04.067 INFO:teuthology.orchestra.run.smithi064.stdout:2177 writesame 0x47960a0 thru 0x47999c8 data_size 0x1(0x3929 bytes) 2024-04-24T00:35:04.076 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:04.070+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:04.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:04.081 INFO:teuthology.orchestra.run.smithi064.stdout:2178 trunc from 0xdc605d4 to 0xda2bd74 2024-04-24T00:35:04.098 INFO:teuthology.orchestra.run.smithi064.stdout:2179 write 0x3661a39 thru 0x36679ab (0x5f73 bytes) 2024-04-24T00:35:04.107 INFO:teuthology.orchestra.run.smithi064.stdout:2180 punch from 0x29dee97 to 0x29e7fc8, (0x9131 bytes) 2024-04-24T00:35:04.107 INFO:teuthology.orchestra.run.smithi064.stdout:2181 punch from 0x7b5b459 to 0x7b64cc7, (0x986e bytes) 2024-04-24T00:35:04.107 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:04.107 INFO:teuthology.orchestra.run.smithi064.stdout:2184 read 0x1a42bc2 thru 0x1a50797 (0xdbd6 bytes) 2024-04-24T00:35:04.108 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:04.108 INFO:teuthology.orchestra.run.smithi064.stdout:2187 writesame 0x4831997 thru 0x48402a8 data_size 0x1(0xe912 bytes) 2024-04-24T00:35:04.169 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:35:04.177 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:04.171+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:04.203 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:04.203 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:04.214 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:04.214 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:04.220 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:04.220 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:04.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:04.231 INFO:teuthology.orchestra.run.smithi064.stdout:2188 writesame 0x69e0f08 thru 0x69eda39 data_size 0x1f(0xcb32 bytes) 2024-04-24T00:35:04.238 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:04.238 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:04.240 INFO:teuthology.orchestra.run.smithi064.stdout:2189 write 0x644625f thru 0x64518ab (0xb64d bytes) 2024-04-24T00:35:04.250 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:04.250 INFO:teuthology.orchestra.run.smithi064.stdout:2194 punch from 0x65c8138 to 0x65c9ab0, (0x1978 bytes) 2024-04-24T00:35:04.250 INFO:teuthology.orchestra.run.smithi064.stdout:2195 write 0x2a4cf64 thru 0x2a5364b (0x66e8 bytes) 2024-04-24T00:35:04.257 INFO:teuthology.orchestra.run.smithi064.stdout:2196 write 0x453362b thru 0x45406c5 (0xd09b bytes) 2024-04-24T00:35:04.258 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:04.258 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:04.265 INFO:teuthology.orchestra.run.smithi064.stdout:2197 read 0x4bed61c thru 0x4beef15 (0x18fa bytes) 2024-04-24T00:35:04.266 INFO:teuthology.orchestra.run.smithi064.stdout:2198 write 0xa14ddfe thru 0xa14e165 (0x368 bytes) 2024-04-24T00:35:04.273 INFO:teuthology.orchestra.run.smithi064.stdout:2199 read 0x82a1e21 thru 0x82a7c9f (0x5e7f bytes) 2024-04-24T00:35:04.273 INFO:teuthology.orchestra.run.smithi064.stdout:2200 writesame 0x2086436 thru 0x20931a9 data_size 0x36(0xcd74 bytes) 2024-04-24T00:35:04.278 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:04.279 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:04.275+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:04.279 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:04.289 INFO:teuthology.orchestra.run.smithi064.stdout:2201 trunc from 0xda2bd74 to 0x449b21f 2024-04-24T00:35:04.301 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:04.301 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:04.307 INFO:teuthology.orchestra.run.smithi064.stdout:2202 read 0x3d194d6 thru 0x3d20f21 (0x7a4c bytes) 2024-04-24T00:35:04.307 INFO:teuthology.orchestra.run.smithi064.stdout:2203 read 0x43f44c9 thru 0x44010b1 (0xcbe9 bytes) 2024-04-24T00:35:04.320 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:04.320 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:04.322 INFO:teuthology.orchestra.run.smithi064.stdout:2204 write 0x9e86475 thru 0x9e8ebb0 (0x873c bytes) 2024-04-24T00:35:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:35:04.324 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:35:04.331 INFO:teuthology.orchestra.run.smithi064.stdout:2205 writesame 0x341bf18 thru 0x342ad72 data_size 0x17(0xee5b bytes) 2024-04-24T00:35:04.339 INFO:teuthology.orchestra.run.smithi064.stdout:2206 read 0x779b235 thru 0x77a4464 (0x9230 bytes) 2024-04-24T00:35:04.349 INFO:teuthology.orchestra.run.smithi064.stdout:2207 write 0xb8fd4d0 thru 0xb9069bb (0x94ec bytes) 2024-04-24T00:35:04.360 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:04.360 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:65754 3.1 e85/59 3:8048459e:::rbd_data.115e8561432.00000000000002e4:head v 85'115, mlcod=85'113)", 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.359057+0000", 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00056519299999999997, 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058252800000000002, 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.359057+0000", 2024-04-24T00:35:04.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.359057+0000", 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.359058+0000", 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.359110+0000", 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2061000000000002e-05 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.359123+0000", 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2281e-05 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.359129+0000", 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9920000000000004e-06 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.359195+0000", 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6624000000000005e-05 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.359221+0000", 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5323000000000001e-05 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:35:04.364 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-04-24T00:35:04.365 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:04.365 INFO:teuthology.orchestra.run.smithi064.stdout:2208 read 0xcd133b thru 0xcdbb0f (0xa7d5 bytes) 2024-04-24T00:35:04.365 INFO:teuthology.orchestra.run.smithi064.stdout:2209 read 0x9952dfb thru 0x9958e98 (0x609e bytes) 2024-04-24T00:35:04.365 INFO:teuthology.orchestra.run.smithi064.stdout:2211 write 0xa3635bf thru 0xa36a02a (0x6a6c bytes) 2024-04-24T00:35:04.373 INFO:teuthology.orchestra.run.smithi064.stdout:2212 read 0x7cd67c7 thru 0x7cdb6f9 (0x4f33 bytes) 2024-04-24T00:35:04.373 INFO:teuthology.orchestra.run.smithi064.stdout:2213 write 0x14fdd63 thru 0x14ff76b (0x1a09 bytes) 2024-04-24T00:35:04.375 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:04.375 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:04.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:04.388 INFO:teuthology.orchestra.run.smithi064.stdout:2217 trunc from 0xb9069bc to 0x28a1812 2024-04-24T00:35:04.388 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:04.382+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:04.425 INFO:teuthology.orchestra.run.smithi064.stdout:2219 write 0x50bdf7b thru 0x50c7c8e (0x9d14 bytes) 2024-04-24T00:35:04.453 INFO:teuthology.orchestra.run.smithi064.stdout:2220 write 0x9366f22 thru 0x936a0ba (0x3199 bytes) 2024-04-24T00:35:04.463 INFO:teuthology.orchestra.run.smithi064.stdout:2221 write 0x85344f7 thru 0x85359a6 (0x14b0 bytes) 2024-04-24T00:35:04.471 INFO:teuthology.orchestra.run.smithi064.stdout:2222 read 0x634b1a5 thru 0x634ed6e (0x3bca bytes) 2024-04-24T00:35:04.472 INFO:teuthology.orchestra.run.smithi064.stdout:2225 writesame 0x40db1c1 thru 0x40e8989 data_size 0x1(0xd7c9 bytes) 2024-04-24T00:35:04.479 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:04.474+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:04.525 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:04.526 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:04.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:04.580 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:04.575+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:04.582 INFO:teuthology.orchestra.run.smithi064.stdout:2226 trunc from 0x936a0bb to 0xa15ab20 2024-04-24T00:35:04.593 INFO:teuthology.orchestra.run.smithi064.stdout:2228 read 0x720f1c7 thru 0x721d94a (0xe784 bytes) 2024-04-24T00:35:04.601 INFO:teuthology.orchestra.run.smithi064.stdout:2229 trunc from 0xa15ab20 to 0x4fc78da 2024-04-24T00:35:04.625 INFO:teuthology.orchestra.run.smithi064.stdout:2231 read 0x339449e thru 0x339e2f7 (0x9e5a bytes) 2024-04-24T00:35:04.640 INFO:teuthology.orchestra.run.smithi064.stdout:2232 write 0x94756c0 thru 0x947790d (0x224e bytes) 2024-04-24T00:35:04.649 INFO:teuthology.orchestra.run.smithi064.stdout:2234 punch from 0x156fe6 to 0x16031f, (0x9339 bytes) 2024-04-24T00:35:04.649 INFO:teuthology.orchestra.run.smithi064.stdout:2235 punch from 0x68e4a90 to 0x68f37d7, (0xed47 bytes) 2024-04-24T00:35:04.660 INFO:teuthology.orchestra.run.smithi064.stdout:2236 trunc from 0x947790e to 0x4f866f 2024-04-24T00:35:04.683 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:04.679+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:04.727 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194887770999998, 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:35:04.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194863857000001, 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.732 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:04.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:04.736 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194836682999998, 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:35:04.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194810578000002, 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:04.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194399597, 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:04.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194379787999999, 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194326728999997, 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194291872999997, 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:04.762 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194270308999997, 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:35:04.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.766 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194240817000001, 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.767 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:35:04.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194219955000001, 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:35:04.771 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.775 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.194197381000002, 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:35:04.779 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.511937883000002, 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.780 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:35:04.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:35:04.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:35:04.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.511586968000003, 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:35:04.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.504698648000002, 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:35:04.791 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:35:04.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44653 3.c e79/56 3:3545ecd3:::rbd_data.115ec7ed39de.0000000000000002:head v 79'268, mlcod=77'266)", 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:04.795 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.8339549269999997, 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022668765, 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:04.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7800000000000002e-07 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892860+0000", 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003115821 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892903+0000", 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3176999999999997e-05 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892926+0000", 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2459999999999998e-05 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.893026+0000", 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010062 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.893059+0000", 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2806999999999999e-05 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.912296+0000", 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019237431999999999 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.912412+0000", 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011597 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44656 3.7 e79/44 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:head v 79'103, mlcod=77'101)", 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.8293721820000002, 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026507441999999999, 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894327+0000", 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1e-07 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898194+0000", 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038672889999999999 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898203+0000", 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6340000000000007e-06 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898208+0000", 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2009999999999998e-06 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898262+0000", 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4466999999999999e-05 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898280+0000", 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7492000000000001e-05 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920764+0000", 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022483974 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920834+0000", 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0174999999999998e-05 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44661 3.17 e79/59 3:eff1c653:::rbd_data.115ec7ed39de.0000000000000005:head v 79'91, mlcod=77'89)", 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.8275048930000004, 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024065629000000002, 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:04.804 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8299999999999996e-07 2024-04-24T00:35:04.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897771+0000", 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015761690000000001 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897786+0000", 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5051999999999999e-05 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897794+0000", 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4689999999999997e-06 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897869+0000", 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.47e-05 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897899+0000", 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9777e-05 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:04.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920181+0000", 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022282560999999999 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920259+0000", 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8218000000000002e-05 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:04.808 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.24641702200000001, 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:35:04.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.236140606, 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:35:04.814 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:35:04.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:35:04.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:04.818 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:04.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:04.820 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:35:04.820 INFO:teuthology.orchestra.run.smithi064.stdout:2237 write 0xc6b31ed thru 0xc6c2976 (0xf78a bytes) 2024-04-24T00:35:04.822 INFO:teuthology.orchestra.run.smithi064.stdout:2240 write 0xe9056af thru 0xe908feb (0x393d bytes) 2024-04-24T00:35:04.822 INFO:teuthology.orchestra.run.smithi064.stdout:2241 write 0xe038e24 thru 0xe04721a (0xe3f7 bytes) 2024-04-24T00:35:04.822 INFO:teuthology.orchestra.run.smithi064.stdout:2242 punch from 0x5332e27 to 0x533c1c6, (0x939f bytes) 2024-04-24T00:35:04.822 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:04.814+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:04.834 INFO:teuthology.orchestra.run.smithi064.stdout:2244 trunc from 0xe908fec to 0x5dd7937 2024-04-24T00:35:04.850 INFO:teuthology.orchestra.run.smithi064.stdout:2245 writesame 0x652ffb thru 0x653141 data_size 0x1(0x147 bytes) 2024-04-24T00:35:04.854 INFO:teuthology.orchestra.run.smithi064.stdout:2249 writesame 0x393bab thru 0x398504 data_size 0x29(0x495a bytes) 2024-04-24T00:35:04.857 INFO:teuthology.orchestra.run.smithi064.stdout:2252 writesame 0xd544bb thru 0xd61bfb data_size 0x5(0xd741 bytes) 2024-04-24T00:35:04.870 INFO:teuthology.orchestra.run.smithi064.stdout:2253 read 0x1ba5ae0 thru 0x1baccab (0x71cc bytes) 2024-04-24T00:35:04.884 INFO:teuthology.orchestra.run.smithi064.stdout:2254 writesame 0x78a0b8d thru 0x78ab43e data_size 0x1a(0xa8b2 bytes) 2024-04-24T00:35:04.884 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:04.899 INFO:teuthology.orchestra.run.smithi064.stdout:2256 writesame 0x930150e thru 0x930e307 data_size 0x5(0xcdfa bytes) 2024-04-24T00:35:04.899 INFO:teuthology.orchestra.run.smithi064.stdout:2258 punch from 0x20de26f to 0x20ea5cd, (0xc35e bytes) 2024-04-24T00:35:04.899 INFO:teuthology.orchestra.run.smithi064.stdout:2259 punch from 0x63fd3a8 to 0x6402714, (0x536c bytes) 2024-04-24T00:35:04.919 INFO:teuthology.orchestra.run.smithi064.stdout:2260 write 0xc3e2f33 thru 0xc3f19f6 (0xeac4 bytes) 2024-04-24T00:35:04.920 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:04.914+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:04.926 INFO:teuthology.orchestra.run.smithi064.stdout:2261 write 0xb69f6e3 thru 0xb6a84d7 (0x8df5 bytes) 2024-04-24T00:35:04.936 INFO:teuthology.orchestra.run.smithi064.stdout:2262 trunc from 0xc3f19f7 to 0xa9ed4c3 2024-04-24T00:35:04.958 INFO:teuthology.orchestra.run.smithi064.stdout:2263 writesame 0x19b52a8 thru 0x19bbad6 data_size 0x1(0x682f bytes) 2024-04-24T00:35:04.964 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:04.964 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:04.964 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:04.964 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:04.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:05.006 INFO:teuthology.orchestra.run.smithi064.stdout:2264 read 0x6bfd688 thru 0x6c03920 (0x6299 bytes) 2024-04-24T00:35:05.006 INFO:teuthology.orchestra.run.smithi064.stdout:2265 trunc from 0xa9ed4c3 to 0xacb70a6 2024-04-24T00:35:05.020 INFO:teuthology.orchestra.run.smithi064.stdout:2267 writesame 0x691f0cd thru 0x6926b90 data_size 0x6(0x7ac4 bytes) 2024-04-24T00:35:05.030 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:05.027+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:05.031 INFO:teuthology.orchestra.run.smithi064.stdout:2268 writesame 0x53dc057 thru 0x53e28ff data_size 0x27(0x68a9 bytes) 2024-04-24T00:35:05.038 INFO:teuthology.orchestra.run.smithi064.stdout:2269 write 0xa56580 thru 0xa58d49 (0x27ca bytes) 2024-04-24T00:35:05.045 INFO:teuthology.orchestra.run.smithi064.stdout:2270 write 0x339fb71 thru 0x33a1a2a (0x1eba bytes) 2024-04-24T00:35:05.050 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.059 INFO:teuthology.orchestra.run.smithi064.stdout:2273 trunc from 0xacb70a6 to 0x7bea995 2024-04-24T00:35:05.071 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.071 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.072 INFO:teuthology.orchestra.run.smithi064.stdout:2277 read 0x5ae84fb thru 0x5aeaf59 (0x2a5f bytes) 2024-04-24T00:35:05.073 INFO:teuthology.orchestra.run.smithi064.stdout:2279 trunc from 0x7bea995 to 0x6f6f8e9 2024-04-24T00:35:05.105 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:05.106 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:05.106 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:05.106 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:05.106 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:05.106 INFO:teuthology.orchestra.run.smithi064.stdout:2280 writesame 0x9d9136a thru 0x9d9f4f0 data_size 0xf(0xe187 bytes) 2024-04-24T00:35:05.106 INFO:teuthology.orchestra.run.smithi064.stdout:2281 write 0x69e41fa thru 0x69ecf80 (0x8d87 bytes) 2024-04-24T00:35:05.115 INFO:teuthology.orchestra.run.smithi064.stdout:2284 read 0x63cf24 thru 0x649f40 (0xd01d bytes) 2024-04-24T00:35:05.116 INFO:teuthology.orchestra.run.smithi064.stdout:2285 read 0x6666240 thru 0x666b212 (0x4fd3 bytes) 2024-04-24T00:35:05.117 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.117 INFO:teuthology.orchestra.run.smithi064.stdout:2289 read 0x1432525 thru 0x1436142 (0x3c1e bytes) 2024-04-24T00:35:05.117 INFO:teuthology.orchestra.run.smithi064.stdout:2291 writesame 0x6f5586f thru 0x6f59a1d data_size 0x3b(0x41af bytes) 2024-04-24T00:35:05.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:05.122 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:05.115+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:05.149 INFO:teuthology.orchestra.run.smithi064.stdout:2292 write 0xd13785b thru 0xd13aafe (0x32a4 bytes) 2024-04-24T00:35:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:2293 read 0xc622e83 thru 0xc631a75 (0xebf3 bytes) 2024-04-24T00:35:05.191 INFO:teuthology.orchestra.run.smithi064.stdout:2294 trunc from 0xd13aaff to 0x973818 2024-04-24T00:35:05.221 INFO:teuthology.orchestra.run.smithi064.stdout:2295 read 0x46c161 thru 0x471f5e (0x5dfe bytes) 2024-04-24T00:35:05.222 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:05.215+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:05.223 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.223 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.244 INFO:teuthology.orchestra.run.smithi064.stdout:2298 writesame 0x37a1cdf thru 0x37a9478 data_size 0x51(0x779a bytes) 2024-04-24T00:35:05.244 INFO:teuthology.orchestra.run.smithi064.stdout:2300 punch from 0x363dc7e to 0x36406c4, (0x2a46 bytes) 2024-04-24T00:35:05.244 INFO:teuthology.orchestra.run.smithi064.stdout:2301 read 0x2a41946 thru 0x2a42cae (0x1369 bytes) 2024-04-24T00:35:05.244 INFO:teuthology.orchestra.run.smithi064.stdout:2304 write 0x1a568a3 thru 0x1a5c269 (0x59c7 bytes) 2024-04-24T00:35:05.254 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.254 INFO:teuthology.orchestra.run.smithi064.stdout:2307 writesame 0x13db998 thru 0x13e70c6 data_size 0x53(0xb72f bytes) 2024-04-24T00:35:05.263 INFO:teuthology.orchestra.run.smithi064.stdout:2308 read 0x6e63ca thru 0x6e87d0 (0x2407 bytes) 2024-04-24T00:35:05.286 INFO:teuthology.orchestra.run.smithi064.stdout:2310 trunc from 0x37a9479 to 0xb8ab334 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.656582389, 2024-04-24T00:35:05.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:35:05.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:35:05.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.656506190000002, 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.307 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:35:05.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:35:05.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.656048016, 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.311 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:05.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.315 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.655990709000001, 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:35:05.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:35:05.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.655731742999997, 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:35:05.320 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:05.324 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.654810271000002, 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.328 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.082995541000003, 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:05.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:05.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:05.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.067466291999999, 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:05.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:05.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.438742972, 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.339 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:05.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.344 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.423707845000003, 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:35:05.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.661209847000002, 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.349 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:05.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.647386889000003, 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:05.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:05.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:05.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.420245703999999, 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.358 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:05.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:05.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:05.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:05.363 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.408456053999998, 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:05.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30704 3.1a 3:5e74e97a:::rbd_data.115e8210e51a.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.044964135000001, 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043351496000000003, 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.368 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30704 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:05.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252358+0000", 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1790000000000001e-06 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252364+0000", 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6549999999999998e-06 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252381+0000", 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6767000000000001e-05 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252390+0000", 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9959999999999999e-06 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252492+0000", 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102563 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252668+0000", 2024-04-24T00:35:05.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017553499999999999 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.254545+0000", 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001877037 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295422+0000", 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040876527000000003 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295513+0000", 2024-04-24T00:35:05.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1838999999999999e-05 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295614+0000", 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010027900000000001 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295708+0000", 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4118999999999998e-05 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.414415989999998, 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.374 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:05.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:05.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.379 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33518 3.b e70/59 ondisk, result = 0)", 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.403092462, 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042187448000000002, 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894230+0000", 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7680000000000001e-06 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894239+0000", 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3319999999999998e-06 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894266+0000", 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049000000000001e-05 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894281+0000", 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5245000000000001e-05 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936247+0000", 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041965833000000001 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936257+0000", 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936415+0000", 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158605 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33734 3.b 3:d4ebe596:::rbd_data.115ec4f9b3a7.000000000000000e:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:05.383 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.778050267000001, 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041668288999999997, 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33734 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519272+0000", 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9870000000000002e-06 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519282+0000", 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0203e-05 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519310+0000", 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7880000000000001e-05 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519324+0000", 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519402+0000", 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7865000000000001e-05 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519481+0000", 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9103000000000005e-05 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.522670+0000", 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003189 2024-04-24T00:35:05.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560812+0000", 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038141427999999998 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560831+0000", 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9820999999999999e-05 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560886+0000", 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5087999999999998e-05 2024-04-24T00:35:05.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560938+0000", 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1993999999999997e-05 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44651 3.6 e79/59 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head v 79'72, mlcod=77'70)", 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.4083428990000009, 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048315631999999997, 2024-04-24T00:35:05.389 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892459+0000", 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034802259999999999 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892475+0000", 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5998000000000002e-05 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892485+0000", 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8679999999999994e-06 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892653+0000", 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016800800000000001 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892721+0000", 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8953000000000002e-05 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937243+0000", 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044521666000000001 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937293+0000", 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0043000000000002e-05 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44659 3.1d e79/53 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head v 79'105, mlcod=77'103)", 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:05.393 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.4008617159999996, 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038358476000000002, 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:05.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896460+0000", 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2399999999999997e-07 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897721+0000", 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012617049999999999 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897744+0000", 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2982000000000001e-05 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897756+0000", 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1559999999999999e-05 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897836+0000", 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0434999999999998e-05 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897871+0000", 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5147000000000001e-05 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934696+0000", 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036824511999999997 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934817+0000", 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121311 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.398 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.398 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:05.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2311 trunc from 0xb8ab334 to 0xed3d08a 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2315 read 0xa4f6b6e thru 0xa4f8b25 (0x1fb8 bytes) 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2316 writesame 0xce437d7 thru 0xce4eff0 data_size 0xf(0xb81a bytes) 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2317 read 0x7befc3c thru 0x7bf8a97 (0x8e5c bytes) 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2318 read 0x11947e9 thru 0x11a1f47 (0xd75f bytes) 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2323 read 0x14fa64b thru 0x15091fd (0xebb3 bytes) 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2324 punch from 0x36c0287 to 0x36c9331, (0x90aa bytes) 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2325 write 0x692f260 thru 0x6937b19 (0x88ba bytes) 2024-04-24T00:35:05.401 INFO:teuthology.orchestra.run.smithi064.stdout:2329 writesame 0x7d96870 thru 0x7d9694d data_size 0x3(0xde bytes) 2024-04-24T00:35:05.402 INFO:teuthology.orchestra.run.smithi064.stdout:2330 writesame 0x64ce838 thru 0x64d3b05 data_size 0x3(0x52ce bytes) 2024-04-24T00:35:05.402 INFO:teuthology.orchestra.run.smithi064.stdout:2331 trunc from 0xed3d08a to 0xd55ecdf 2024-04-24T00:35:05.402 INFO:teuthology.orchestra.run.smithi064.stdout:2332 write 0xd256f14 thru 0xd25ed7b (0x7e68 bytes) 2024-04-24T00:35:05.402 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:05.402 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:05.394+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi064.stdout:2336 trunc from 0xd55ecdf to 0x2f434d7 2024-04-24T00:35:05.447 INFO:teuthology.orchestra.run.smithi064.stdout:2338 punch from 0x1916f50 to 0x192620d, (0xf2bd bytes) 2024-04-24T00:35:05.447 INFO:teuthology.orchestra.run.smithi064.stdout:2339 punch from 0x21f4dfb to 0x21fa725, (0x592a bytes) 2024-04-24T00:35:05.447 INFO:teuthology.orchestra.run.smithi064.stdout:2340 clone 15 order 20 su 131072 sc 10 2024-04-24T00:35:05.500 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:05.495+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:05.574 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:05.575 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:05.575 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:05.575 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:05.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:05.635 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:05.631+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:05.702 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:05.695+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:05.762 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:05.762 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:05.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:05.822 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:05.819+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:05.904 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:05.899+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:05.953 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:05.953 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:05.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:05.953 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:05.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.239243020000004, 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.954 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.239145094999998, 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:35:05.959 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:35:05.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:35:05.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:35:05.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.879765562999999, 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.966 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:35:05.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:35:05.971 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30133 3.1f e66/59 ondisk, result = 0)", 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.867172016000001, 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060466079999999998, 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080440+0000", 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080446+0000", 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6699999999999999e-06 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080465+0000", 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080476+0000", 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0937999999999999e-05 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140869+0000", 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060392759999999997 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140882+0000", 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3131e-05 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140905+0000", 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2642000000000001e-05 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30705 3.2 3:45c09e0c:::rbd_data.115e8210e51a.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.695159102000002, 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063528001000000001, 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.977 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30705 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252455+0000", 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.303e-06 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252477+0000", 2024-04-24T00:35:05.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1970000000000001e-05 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252515+0000", 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7616e-05 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252536+0000", 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376999999999999e-05 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252696+0000", 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160192 2024-04-24T00:35:05.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252945+0000", 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024906400000000001 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.256753+0000", 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038072129999999998 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315658+0000", 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058905258000000002 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315790+0000", 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013223999999999999 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315894+0000", 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103886 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315980+0000", 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5882000000000002e-05 2024-04-24T00:35:05.982 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30705 3.2 e66/53 ondisk, result = 0)", 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.679946998999998, 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048330598000000002, 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267667+0000", 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.96e-06 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267683+0000", 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6336000000000001e-05 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267730+0000", 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7079000000000001e-05 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267757+0000", 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6364e-05 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315724+0000", 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047967018 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315742+0000", 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7935999999999999e-05 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315995+0000", 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025290499999999999 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33506 3.c 3:3787ff34:::rbd_data.115ec4f9b3a7.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.149562958000001, 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061037313000000003, 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33506 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798052+0000", 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798067+0000", 2024-04-24T00:35:05.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5160000000000001e-05 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798107+0000", 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9946e-05 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798130+0000", 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2637e-05 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798238+0000", 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010841299999999999 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798344+0000", 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010631199999999999 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.801407+0000", 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030623600000000001 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858616+0000", 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057209342000000003 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858879+0000", 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000262952 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858992+0000", 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113429 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859085+0000", 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2925999999999998e-05 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.991 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33506 3.c e70/56 ondisk, result = 0)", 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.139930191000001, 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051419185999999999, 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807682+0000", 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807687+0000", 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807704+0000", 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7424000000000001e-05 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807714+0000", 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3400000000000004e-06 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858714+0000", 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051000248999999998 2024-04-24T00:35:05.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858732+0000", 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7995999999999998e-05 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859100+0000", 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036787400000000002 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35694 3.1b 3:dcf0eb61:::rbd_data.115ec5833ea9.00000000000000aa:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.02811434, 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049399446, 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35694 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:05.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919498+0000", 2024-04-24T00:35:05.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919504+0000", 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919520+0000", 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5908000000000001e-05 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919532+0000", 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2964e-05 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919630+0000", 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7096999999999995e-05 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919734+0000", 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104128 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:05.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.922056+0000", 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023219149999999999 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968775+0000", 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046719274999999998 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968817+0000", 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-05 2024-04-24T00:35:06.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968858+0000", 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1440999999999998e-05 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968896+0000", 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7741000000000003e-05 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.001 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0567815760000006, 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:06.002 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:35:06.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:35:06.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44664 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 524288~131072 [fadvise_sequential+fadvise_nocache],write 1179648~1048576 [fadvise_sequential+fadvise_nocache],write 10616832~2048 [fadvise_sequential+fadvise_nocache],write 10627072~4096 [fadvise_sequential+fadvise_nocache],write 10639360~4096 [fadvise_sequential+fadvise_nocache],write 10651648~4096 [fadvise_sequential+fadvise_nocache],write 10663936~4096 [fadvise_sequential+fadvise_nocache],write 10676224~4096 [fadvise_sequential+fadvise_nocache],write 10688512~4096 [fadvise_sequential+fadvise_nocache],write 10700800~4096 [fadvise_sequential+fadvise_nocache],write 10713088~4096 [fadvise_sequential+fadvise_nocache],write 10725376~4096 [fadvise_sequential+fadvise_nocache],write 10737664~4096 [fadvise_sequential+fadvise_nocache],write 10754048~4096 [fadvise_sequential+fadvise_nocache],write 10766336~4096 [fadvise_sequential+fadvise_nocache],write 10778624~4096 [fadvise_sequential+fadvise_nocache],write 10790912~4096 [fadvise_sequential+fadvise_nocache],write 10803200~4096 [fadvise_sequential+fadvise_nocache],write 10815488~4096 [fadvise_sequential+fadvise_nocache],write 10827776~4096 [fadvise_sequential+fadvise_nocache],write 10840064~4096 [fadvise_sequential+fadvise_nocache],write 10852352~4096 [fadvise_sequential+fadvise_nocache],write 10864640~4096 [fadvise_sequential+fadvise_nocache],write 10876928~2048 [fadvise_sequential+fadvise_nocache],write 10881024~4096 [fadvise_sequential+fadvise_nocache],write 10893312~4096 [fadvise_sequential+fadvise_nocache],write 10905600~4096 [fadvise_sequential+fadvise_nocache],write 10917888~4096 [fadvise_sequential+fadvise_nocache],write 10930176~4096 [fadvise_sequential+fadvise_nocache],write 10942464~4096 [fadvise_sequential+fadvise_nocache],write 10954752~4096 [fadvise_sequential+fadvise_nocache],write 10967040~4096 [fadvise_sequential+fadvise_nocache],write 10979328~4096 [fadvise_sequential+fadvise_nocache],write 10991616~4096 [fadvise_sequential+fadvise_nocache],write 11003904~4096 [fadvise_sequential+fadvise_nocache],write 11010048~2048 [fadvise_sequential+fadvise_nocache],write 11020288~4096 [fadvise_sequential+fadvise_nocache],write 11032576~4096 [fadvise_sequential+fadvise_nocache],write 11044864~4096 [fadvise_sequential+fadvise_nocache],write 11057152~4096 [fadvise_sequential+fadvise_nocache],write 11069440~4096 [fadvise_sequential+fadvise_nocache],write 11081728~4096 [fadvise_sequential+fadvise_nocache],write 11094016~4096 [fadvise_sequential+fadvise_nocache],write 11106304~4096 [fadvise_sequential+fadvise_nocache],write 11118592~4096 [fadvise_sequential+fadvise_nocache],write 11130880~4096 [fadvise_sequential+fadvise_nocache],write 14417920~4096 [fadvise_sequential+fadvise_nocache],write 14426112~8192 [fadvise_sequential+fadvise_nocache],write 14438400~8192 [fadvise_sequential+fadvise_nocache],write 14450688~8192 [fadvise_sequential+fadvise_nocache],write 14462976~8192 [fadvise_sequential+fadvise_nocache],write 14475264~8192 [fadvise_sequential+fadvise_nocache],write 14487552~8192 [fadvise_sequential+fadvise_nocache],write 14499840~8192 [fadvise_sequential+fadvise_nocache],write 14512128~8192 [fadvise_sequential+fadvise_nocache],write 14524416~8192 [fadvise_sequential+fadvise_nocache],write 14536704~8192 [fadvise_sequential+fadvise_nocache],write 14553088~8192 [fadvise_sequential+fadvise_nocache],write 14565376~8192 [fadvise_sequential+fadvise_nocache],write 14577664~8192 [fadvise_sequential+fadvise_nocache],write 14589952~8192 [fadvise_sequential+fadvise_nocache],write 14602240~8192 [fadvise_sequential+fadvise_nocache],write 14614528~8192 [fadvise_sequential+fadvise_nocache],write 14626816~8192 [fadvise_sequential+fadvise_nocache],write 14639104~8192 [fadvise_sequential+fadvise_nocache],write 14651392~8192 [fadvise_sequential+fadvise_nocache],write 14663680~8192 [fadvise_sequential+fadvise_nocache],write 14675968~4096 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0447082549999998, 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768734, 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:06.008 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44664 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902904+0000", 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5179999999999999e-06 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.904804+0000", 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018996390000000001 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905393+0000", 2024-04-24T00:35:06.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058872600000000003 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905408+0000", 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5073e-05 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905482+0000", 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4284999999999998e-05 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905785+0000", 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030266100000000002 2024-04-24T00:35:06.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.907249+0000", 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001464899 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.932396+0000", 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025146379 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958308+0000", 2024-04-24T00:35:06.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025912418999999999 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958527+0000", 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218297 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958671+0000", 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144838 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.013 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46463 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:1d [read 360448~8192 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3495315779999997, 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042258337999999999, 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46463 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.014 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598080+0000", 2024-04-24T00:35:06.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598083+0000", 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6740000000000001e-06 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598091+0000", 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1089999999999998e-06 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598097+0000", 2024-04-24T00:35:06.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5370000000000003e-06 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.635329+0000", 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037231775000000002 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.635418+0000", 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8629999999999997e-05 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640338+0000", 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049199730000000002 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46464 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3495056549999997, 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042285080000000003, 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:06.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46464 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598106+0000", 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7700000000000003e-07 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598109+0000", 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8080000000000001e-06 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598117+0000", 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2430000000000005e-06 2024-04-24T00:35:06.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598124+0000", 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6000000000000003e-06 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640379+0000", 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042255297999999997 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640390+0000", 2024-04-24T00:35:06.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1253999999999999e-05 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46489 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598717+0000", 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3488935040000003, 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041719732000000002, 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:06.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46489 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598717+0000", 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598717+0000", 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598718+0000", 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4e-07 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598732+0000", 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3906e-05 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598749+0000", 2024-04-24T00:35:06.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6908000000000001e-05 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598761+0000", 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2232000000000001e-05 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640427+0000", 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041665617000000002 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640437+0000", 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0229e-05 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46490 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598775+0000", 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3488358500000004, 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041708658000000003, 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:06.026 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46490 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598775+0000", 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598775+0000", 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598776+0000", 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598781+0000", 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6689999999999997e-06 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598795+0000", 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4694e-05 2024-04-24T00:35:06.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598807+0000", 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1765e-05 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640474+0000", 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041666630000000003 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640484+0000", 2024-04-24T00:35:06.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0000000000000001e-05 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46491 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598820+0000", 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3487909079999998, 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041702850999999999, 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.030 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46491 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598820+0000", 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598820+0000", 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598821+0000", 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.026e-06 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598826+0000", 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4889999999999997e-06 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598840+0000", 2024-04-24T00:35:06.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4528e-05 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598852+0000", 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2106000000000001e-05 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640516+0000", 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041664113000000003 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640523+0000", 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5889999999999999e-06 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46492 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598976+0000", 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3486348059999997, 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041582613999999997, 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.035 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46492 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598976+0000", 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598976+0000", 2024-04-24T00:35:06.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598978+0000", 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598986+0000", 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6909999999999997e-06 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599005+0000", 2024-04-24T00:35:06.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9777000000000001e-05 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599020+0000", 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5197e-05 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640552+0000", 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041532001999999998 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640559+0000", 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2099999999999998e-06 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46493 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.599042+0000", 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.348568427, 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041549871000000002, 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.039 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46493 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599042+0000", 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599042+0000", 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599044+0000", 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599050+0000", 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2349999999999996e-06 2024-04-24T00:35:06.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599067+0000", 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7039000000000001e-05 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599081+0000", 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4154999999999999e-05 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640585+0000", 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041504172999999998 2024-04-24T00:35:06.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640592+0000", 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9489999999999999e-06 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:06.043 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.4690214369999999, 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:35:06.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:06.049 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.4437528310000001, 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:06.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:06.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:06.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:06.053 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:06.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:06.056 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:06.051+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:06.156 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:06.151+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:06.226 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:06.226 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:06.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:06.258 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:06.254+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:06.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:06.359 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:06.355+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:06.400 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:06.401 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:06.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:06.460 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:06.455+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:06.463 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:35:06.481 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:06.481 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:06.498 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:06.498 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:06.515 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:06.515 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:06.534 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:06.555 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:06.555 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:06.560 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:06.555+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:06.576 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:06.576 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:06.584 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:06.584 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:06.598 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:06.598 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:06.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:35:06.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:35:06.686 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:06.683+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:06.762 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:06.758+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:06.765 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:06.765 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:06.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:06.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:35:06.866 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:06.863+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:06.958 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:06.958 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:06.963 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:06.959+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:06.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:07.063 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:07.059+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:07.137 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:07.138 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:07.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:07.197 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:07.194+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:07.266 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:07.259+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:07.271 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:07.271 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:07.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:07.367 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:07.363+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:07.457 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:07.457 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:07.468 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:07.463+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:07.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:07.569 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:07.563+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:07.589 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:07.589 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:07.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:07.669 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:07.666+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:07.725 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:07.725 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:07.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:07.781 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:07.778+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:07.784 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:35:07.871 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:07.867+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:07.894 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-04-24T00:35:07.914 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:07.914 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:07.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:07.972 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:07.967+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:08.057 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:08.057 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:08.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:08.073 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:08.067+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:08.174 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:08.170+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:08.190 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-04-24T00:35:08.190 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:35:08.191 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-24T00:35:08.191 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-24T00:35:08.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-24T00:35:08.194 INFO:tasks.ceph.osd.0:Started 2024-04-24T00:35:08.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:08.194 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:08.194 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:08.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:08.219 INFO:teuthology.orchestra.run.smithi064.stdout:2341 read 0x11f62de thru 0x11f88db (0x25fe bytes) 2024-04-24T00:35:08.243 INFO:teuthology.orchestra.run.smithi064.stdout:2344 writesame 0x3c2872f thru 0x3c31834 data_size 0x13(0x9106 bytes) 2024-04-24T00:35:08.243 INFO:teuthology.orchestra.run.smithi064.stdout:2345 read 0x157759c thru 0x15801c5 (0x8c2a bytes) 2024-04-24T00:35:08.243 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.243 INFO:teuthology.orchestra.run.smithi064.stdout:2348 read 0x8c75cd thru 0x8cb2c9 (0x3cfd bytes) 2024-04-24T00:35:08.244 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.244 INFO:teuthology.orchestra.run.smithi064.stdout:2350 read 0x158c1d6 thru 0x159914e (0xcf79 bytes) 2024-04-24T00:35:08.258 INFO:teuthology.orchestra.run.smithi064.stdout:2352 writesame 0x50a223c thru 0x50ab690 data_size 0x17(0x9455 bytes) 2024-04-24T00:35:08.259 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.275 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:08.270+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:08.280 INFO:teuthology.orchestra.run.smithi064.stdout:2354 write 0xb57e8fd thru 0xb58dcb3 (0xf3b7 bytes) 2024-04-24T00:35:08.289 INFO:teuthology.orchestra.run.smithi064.stdout:2355 write 0x2f6ac97 thru 0x2f6cbf3 (0x1f5d bytes) 2024-04-24T00:35:08.295 INFO:teuthology.orchestra.run.smithi064.stdout:2356 writesame 0x5aa21c3 thru 0x5aa9d81 data_size 0x1(0x7bbf bytes) 2024-04-24T00:35:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-04-24T00:35:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:config diff get 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:config help [] 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-04-24T00:35:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-04-24T00:35:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T00:35:08.318 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-24T00:35:08.344 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:08.344 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:08.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:08.401 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:08.398+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:2357 writesame 0xb0419ee thru 0xb045895 data_size 0x14(0x3ea8 bytes) 2024-04-24T00:35:08.410 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.411 INFO:teuthology.orchestra.run.smithi064.stdout:2359 write 0x27e4160 thru 0x27e468a (0x52b bytes) 2024-04-24T00:35:08.420 INFO:teuthology.orchestra.run.smithi064.stdout:2360 trunc from 0xb58dcb4 to 0x968e03c 2024-04-24T00:35:08.442 INFO:teuthology.orchestra.run.smithi064.stdout:2361 write 0x92f044c thru 0x92f0b1a (0x6cf bytes) 2024-04-24T00:35:08.450 INFO:teuthology.orchestra.run.smithi064.stdout:2362 read 0x304c3aa thru 0x3050d0b (0x4962 bytes) 2024-04-24T00:35:08.451 INFO:teuthology.orchestra.run.smithi064.stdout:2363 write 0x6500738 thru 0x6507c97 (0x7560 bytes) 2024-04-24T00:35:08.457 INFO:teuthology.orchestra.run.smithi064.stdout:2365 punch from 0xeaf2f0 to 0xeb736f, (0x807f bytes) 2024-04-24T00:35:08.457 INFO:teuthology.orchestra.run.smithi064.stdout:2366 read 0x493c1c8 thru 0x493cdf5 (0xc2e bytes) 2024-04-24T00:35:08.457 INFO:teuthology.orchestra.run.smithi064.stdout:2367 write 0x4b5ed95 thru 0x4b6c0a8 (0xd314 bytes) 2024-04-24T00:35:08.472 INFO:teuthology.orchestra.run.smithi064.stdout:2368 read 0x2b8f10 thru 0x2bd283 (0x4374 bytes) 2024-04-24T00:35:08.474 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.477 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:08.471+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:08.500 INFO:teuthology.orchestra.run.smithi064.stdout:2370 write 0xdba1a1f thru 0xdbb1429 (0xfa0b bytes) 2024-04-24T00:35:08.509 INFO:teuthology.orchestra.run.smithi064.stdout:2372 trunc from 0xdbb142a to 0xe2df843 2024-04-24T00:35:08.523 INFO:teuthology.orchestra.run.smithi064.stdout:2373 punch from 0x1ab078e to 0x1ab3333, (0x2ba5 bytes) 2024-04-24T00:35:08.523 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.523 INFO:teuthology.orchestra.run.smithi064.stdout:2376 punch from 0x599fa9 to 0x59d829, (0x3880 bytes) 2024-04-24T00:35:08.524 INFO:teuthology.orchestra.run.smithi064.stdout:2377 write 0x6e8afb9 thru 0x6e97993 (0xc9db bytes) 2024-04-24T00:35:08.535 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.535 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:08.535 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:08.544 INFO:teuthology.orchestra.run.smithi064.stdout:2379 trunc from 0xe2df843 to 0x8c2230d 2024-04-24T00:35:08.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:08.580 INFO:teuthology.orchestra.run.smithi064.stdout:2380 trunc from 0x8c2230d to 0xb48f6c6 2024-04-24T00:35:08.581 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:08.574+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:08.590 INFO:teuthology.orchestra.run.smithi064.stdout:2382 writesame 0x8cdf60a thru 0x8ce8109 data_size 0x40(0x8b00 bytes) 2024-04-24T00:35:08.598 INFO:teuthology.orchestra.run.smithi064.stdout:2383 punch from 0x8706d1b to 0x871200d, (0xb2f2 bytes) 2024-04-24T00:35:08.598 INFO:teuthology.orchestra.run.smithi064.stdout:2384 write 0x81622a thru 0x8201bb (0x9f92 bytes) 2024-04-24T00:35:08.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:08.619 INFO:teuthology.orchestra.run.smithi064.stdout:2385 trunc from 0xb48f6c6 to 0x1cfb710 2024-04-24T00:35:08.646 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.646 INFO:teuthology.orchestra.run.smithi064.stdout:2387 punch from 0xd80fda to 0xd856ce, (0x46f4 bytes) 2024-04-24T00:35:08.669 INFO:teuthology.orchestra.run.smithi064.stdout:2388 writesame 0x8762371 thru 0x877021a data_size 0xb(0xdeaa bytes) 2024-04-24T00:35:08.669 INFO:teuthology.orchestra.run.smithi064.stdout:2389 read 0x72cd73f thru 0x72d1201 (0x3ac3 bytes) 2024-04-24T00:35:08.679 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:08.675+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:08.681 INFO:teuthology.orchestra.run.smithi064.stdout:2390 trunc from 0x877021b to 0x14fa99c 2024-04-24T00:35:08.702 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:08.702 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:08.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:08.726 INFO:teuthology.orchestra.run.smithi064.stdout:2393 trunc from 0x14fa99c to 0xd6563a1 2024-04-24T00:35:08.735 INFO:teuthology.orchestra.run.smithi064.stdout:2394 read 0xd181c6c thru 0xd18730c (0x56a1 bytes) 2024-04-24T00:35:08.735 INFO:teuthology.orchestra.run.smithi064.stdout:2395 write 0x10428c2 thru 0x10473c3 (0x4b02 bytes) 2024-04-24T00:35:08.740 INFO:teuthology.orchestra.run.smithi064.stdout:2397 write 0x4267416 thru 0x426f45e (0x8049 bytes) 2024-04-24T00:35:08.746 INFO:teuthology.orchestra.run.smithi064.stdout:2399 read 0x7952cf9 thru 0x795db87 (0xae8f bytes) 2024-04-24T00:35:08.746 INFO:teuthology.orchestra.run.smithi064.stdout:2400 read 0x6f859ae thru 0x6f85b15 (0x168 bytes) 2024-04-24T00:35:08.758 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:35:08.762 INFO:teuthology.orchestra.run.smithi064.stdout:2401 trunc from 0xd6563a1 to 0x2dadb63 2024-04-24T00:35:08.777 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:08.777 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:08.785 INFO:teuthology.orchestra.run.smithi064.stdout:2402 trunc from 0x2dadb63 to 0x23ef84 2024-04-24T00:35:08.795 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:08.795 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:08.802 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:08.799+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:08.813 INFO:teuthology.orchestra.run.smithi064.stdout:2403 trunc from 0x23ef84 to 0x603cade 2024-04-24T00:35:08.814 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:08.815 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:08.821 INFO:teuthology.orchestra.run.smithi064.stdout:2406 punch from 0xeee2a6 to 0xefc5dd, (0xe337 bytes) 2024-04-24T00:35:08.821 INFO:teuthology.orchestra.run.smithi064.stdout:2407 punch from 0x5e0b6da to 0x5e136e8, (0x800e bytes) 2024-04-24T00:35:08.821 INFO:teuthology.orchestra.run.smithi064.stdout:2408 punch from 0x3444bc6 to 0x3450adb, (0xbf15 bytes) 2024-04-24T00:35:08.821 INFO:teuthology.orchestra.run.smithi064.stdout:2409 punch from 0x123ced8 to 0x1249fd0, (0xd0f8 bytes) 2024-04-24T00:35:08.821 INFO:teuthology.orchestra.run.smithi064.stdout:2411 punch from 0x3715afa to 0x3716905, (0xe0b bytes) 2024-04-24T00:35:08.821 INFO:teuthology.orchestra.run.smithi064.stdout:2412 read 0x1837929 thru 0x184446d (0xcb45 bytes) 2024-04-24T00:35:08.821 INFO:teuthology.orchestra.run.smithi064.stdout:2413 write 0x45f26e9 thru 0x45f3c1f (0x1537 bytes) 2024-04-24T00:35:08.826 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:08.822+0000 7f13038ff7c0 -1 Falling back to public interface 2024-04-24T00:35:08.833 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:08.833 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:08.834 INFO:teuthology.orchestra.run.smithi064.stdout:2414 trunc from 0x603cade to 0xd700c02 2024-04-24T00:35:08.841 INFO:teuthology.orchestra.run.smithi064.stdout:2415 write 0x79ba87 thru 0x7a867e (0xcbf8 bytes) 2024-04-24T00:35:08.844 INFO:teuthology.orchestra.run.smithi064.stdout:2416 writesame 0x4eebae6 thru 0x4ef181a data_size 0x1(0x5d35 bytes) 2024-04-24T00:35:08.851 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:08.852 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:08.870 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:08.880 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:08.875+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:08.888 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:08.888 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:08.888 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:08.888 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:08.889 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:08.889 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:08.889 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:35:08.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:35:08.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:08.906 INFO:teuthology.orchestra.run.smithi064.stdout:2418 write 0x4b9f482 thru 0x4ba4111 (0x4c90 bytes) 2024-04-24T00:35:08.910 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.910 INFO:teuthology.orchestra.run.smithi064.stdout:2420 read 0xa20cb2e thru 0xa20cdcd (0x2a0 bytes) 2024-04-24T00:35:08.910 INFO:teuthology.orchestra.run.smithi064.stdout:2421 writesame 0x209a32a thru 0x20a2173 data_size 0x6a(0x7e4a bytes) 2024-04-24T00:35:08.918 INFO:teuthology.orchestra.run.smithi064.stdout:2423 trunc from 0xd700c02 to 0xe7f09fe 2024-04-24T00:35:08.926 INFO:teuthology.orchestra.run.smithi064.stdout:2424 write 0x54db4bf thru 0x54e2b82 (0x76c4 bytes) 2024-04-24T00:35:08.931 INFO:teuthology.orchestra.run.smithi064.stdout:2425 write 0x1a8a436 thru 0x1a91802 (0x73cd bytes) 2024-04-24T00:35:08.937 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:08.937 INFO:teuthology.orchestra.run.smithi064.stdout:2429 write 0x9897307 thru 0x989bec1 (0x4bbb bytes) 2024-04-24T00:35:08.946 INFO:teuthology.orchestra.run.smithi064.stdout:2430 trunc from 0xe7f09fe to 0xba0afa0 2024-04-24T00:35:08.966 INFO:teuthology.orchestra.run.smithi064.stdout:2431 write 0xc01ac73 thru 0xc025733 (0xaac1 bytes) 2024-04-24T00:35:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:2433 punch from 0xb01c5a5 to 0xb0219f2, (0x544d bytes) 2024-04-24T00:35:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:2437 punch from 0x3907bc2 to 0x391710b, (0xf549 bytes) 2024-04-24T00:35:08.981 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:08.978+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:08.994 INFO:teuthology.orchestra.run.smithi064.stdout:2438 trunc from 0xc025734 to 0x453e24a 2024-04-24T00:35:09.016 INFO:teuthology.orchestra.run.smithi064.stdout:2439 write 0xc4b4bb thru 0xc551b9 (0x9cff bytes) 2024-04-24T00:35:09.047 INFO:teuthology.orchestra.run.smithi064.stdout:2440 trunc from 0x453e24a to 0xcc3b6d0 2024-04-24T00:35:09.061 INFO:teuthology.orchestra.run.smithi064.stdout:2441 punch from 0x89018e3 to 0x8909471, (0x7b8e bytes) 2024-04-24T00:35:09.068 INFO:teuthology.orchestra.run.smithi064.stdout:2443 trunc from 0xcc3b6d0 to 0xa43d7f5 2024-04-24T00:35:09.082 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:09.078+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.438922491, 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.083 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:35:09.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:35:09.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.438846292000001, 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.438388117999999, 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:35:09.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.438330811, 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.438071845000003, 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.101 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:09.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:35:09.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.437150373000001, 2024-04-24T00:35:09.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.865335643000002, 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.110 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:09.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:09.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.849806393999998, 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:09.116 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:09.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:09.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:09.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.221083073999999, 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:09.123 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:09.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:09.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.206047947000002, 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:35:09.129 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:09.133 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.443549949000001, 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:09.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:09.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:09.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.429726991000003, 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.202585805999998, 2024-04-24T00:35:09.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.190796156000001, 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30704 3.1a 3:5e74e97a:::rbd_data.115e8210e51a.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.827304237, 2024-04-24T00:35:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043351496000000003, 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30704 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252358+0000", 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1790000000000001e-06 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252364+0000", 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6549999999999998e-06 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252381+0000", 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6767000000000001e-05 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252390+0000", 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9959999999999999e-06 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252492+0000", 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102563 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252668+0000", 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017553499999999999 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.254545+0000", 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001877037 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295422+0000", 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040876527000000003 2024-04-24T00:35:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295513+0000", 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1838999999999999e-05 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295614+0000", 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010027900000000001 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295708+0000", 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4118999999999998e-05 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.196756092000001, 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33518 3.b e70/59 ondisk, result = 0)", 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.185432563999999, 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042187448000000002, 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894230+0000", 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7680000000000001e-06 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894239+0000", 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3319999999999998e-06 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894266+0000", 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049000000000001e-05 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894281+0000", 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5245000000000001e-05 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936247+0000", 2024-04-24T00:35:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041965833000000001 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936257+0000", 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936415+0000", 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158605 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33734 3.b 3:d4ebe596:::rbd_data.115ec4f9b3a7.000000000000000e:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.560390369, 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041668288999999997, 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33734 2024-04-24T00:35:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519272+0000", 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9870000000000002e-06 2024-04-24T00:35:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519282+0000", 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0203e-05 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519310+0000", 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7880000000000001e-05 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519324+0000", 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:35:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519402+0000", 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7865000000000001e-05 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519481+0000", 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9103000000000005e-05 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.522670+0000", 2024-04-24T00:35:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003189 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560812+0000", 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038141427999999998 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560831+0000", 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9820999999999999e-05 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560886+0000", 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5087999999999998e-05 2024-04-24T00:35:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560938+0000", 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1993999999999997e-05 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44651 3.6 e79/59 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head v 79'72, mlcod=77'70)", 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.190683001, 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048315631999999997, 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892459+0000", 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034802259999999999 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892475+0000", 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5998000000000002e-05 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892485+0000", 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8679999999999994e-06 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892653+0000", 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016800800000000001 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892721+0000", 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8953000000000002e-05 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937243+0000", 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044521666000000001 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937293+0000", 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0043000000000002e-05 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44659 3.1d e79/53 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head v 79'105, mlcod=77'103)", 2024-04-24T00:35:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.183201818000001, 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038358476000000002, 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896460+0000", 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2399999999999997e-07 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897721+0000", 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012617049999999999 2024-04-24T00:35:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897744+0000", 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2982000000000001e-05 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897756+0000", 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1559999999999999e-05 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897836+0000", 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0434999999999998e-05 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897871+0000", 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5147000000000001e-05 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934696+0000", 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036824511999999997 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934817+0000", 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121311 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.182 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:09.183 INFO:teuthology.orchestra.run.smithi064.stdout:2444 write 0xa4a8ded thru 0xa4a96ac (0x8c0 bytes) 2024-04-24T00:35:09.184 INFO:teuthology.orchestra.run.smithi064.stdout:2445 write 0xd6fb8da thru 0xd707bc0 (0xc2e7 bytes) 2024-04-24T00:35:09.184 INFO:teuthology.orchestra.run.smithi064.stdout:2446 write 0x8243856 thru 0x824b7a7 (0x7f52 bytes) 2024-04-24T00:35:09.184 INFO:teuthology.orchestra.run.smithi064.stdout:2448 writesame 0x1aaab34 thru 0x1ab8c51 data_size 0x22(0xe11e bytes) 2024-04-24T00:35:09.184 INFO:teuthology.orchestra.run.smithi064.stdout:2449 trunc from 0xd707bc1 to 0x3ba32bb 2024-04-24T00:35:09.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:09.185 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:09.179+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:09.203 INFO:teuthology.orchestra.run.smithi064.stdout:2450 trunc from 0x3ba32bb to 0xe2780b 2024-04-24T00:35:09.235 INFO:teuthology.orchestra.run.smithi064.stdout:2451 punch from 0x5b9a4b to 0x5bbef0, (0x24a5 bytes) 2024-04-24T00:35:09.235 INFO:teuthology.orchestra.run.smithi064.stdout:2452 read 0xafbfb9 thru 0xaff1a5 (0x31ed bytes) 2024-04-24T00:35:09.235 INFO:teuthology.orchestra.run.smithi064.stdout:2453 read 0xad70f4 thru 0xae0a13 (0x9920 bytes) 2024-04-24T00:35:09.235 INFO:teuthology.orchestra.run.smithi064.stdout:2454 read 0x1ca27f thru 0x1cb59f (0x1321 bytes) 2024-04-24T00:35:09.251 INFO:teuthology.orchestra.run.smithi064.stdout:2455 write 0xf6e468 thru 0xf75243 (0x6ddc bytes) 2024-04-24T00:35:09.259 INFO:teuthology.orchestra.run.smithi064.stdout:2458 read 0x4183e8 thru 0x426f6a (0xeb83 bytes) 2024-04-24T00:35:09.284 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:09.278+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:09.291 INFO:teuthology.orchestra.run.smithi064.stdout:2459 write 0x5e9a660 thru 0x5ea4576 (0x9f17 bytes) 2024-04-24T00:35:09.301 INFO:teuthology.orchestra.run.smithi064.stdout:2460 punch from 0x4f12516 to 0x4f19ca4, (0x778e bytes) 2024-04-24T00:35:09.310 INFO:teuthology.orchestra.run.smithi064.stdout:2463 trunc from 0x5ea4577 to 0x92f13f2 2024-04-24T00:35:09.325 INFO:teuthology.orchestra.run.smithi064.stdout:2464 writesame 0x481ef53 thru 0x4822f45 data_size 0x33(0x3ff3 bytes) 2024-04-24T00:35:09.336 INFO:teuthology.orchestra.run.smithi064.stdout:2466 punch from 0x38285e3 to 0x383354c, (0xaf69 bytes) 2024-04-24T00:35:09.336 INFO:teuthology.orchestra.run.smithi064.stdout:2468 read 0x768405 thru 0x76a3ee (0x1fea bytes) 2024-04-24T00:35:09.352 INFO:teuthology.orchestra.run.smithi064.stdout:2471 trunc from 0x92f13f2 to 0x377cd58 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:67503 3.c 3:32d64c95:::rbd_object_map.115e1da9b58d:head [call lock.assert_locked in=44b,call rbd.object_map_update in=39b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:09.359928+0000", 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00054156400000000002, 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00061547000000000001, 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.362 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 67503 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:09.359928+0000", 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:09.359928+0000", 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:09.359929+0000", 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.093e-06 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:09.359935+0000", 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5949999999999998e-06 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:09.359950+0000", 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5265999999999999e-05 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:09.359959+0000", 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6750000000000008e-06 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:09.360056+0000", 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6642000000000001e-05 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:09.360211+0000", 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000155415 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:35:09.366 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:09.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:09.380 INFO:teuthology.orchestra.run.smithi064.stdout:2472 write 0x45a588 thru 0x46581b (0xb294 bytes) 2024-04-24T00:35:09.385 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:09.378+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:09.398 INFO:teuthology.orchestra.run.smithi064.stdout:2474 trunc from 0x377cd58 to 0x7ebcf52 2024-04-24T00:35:09.409 INFO:teuthology.orchestra.run.smithi064.stdout:2476 read 0x52ce6a7 thru 0x52d8ef5 (0xa84f bytes) 2024-04-24T00:35:09.427 INFO:teuthology.orchestra.run.smithi064.stdout:2477 write 0xa8494f5 thru 0xa84bea5 (0x29b1 bytes) 2024-04-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi064.stdout:2478 read 0x8a790c8 thru 0x8a797cf (0x708 bytes) 2024-04-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi064.stdout:2480 trunc from 0xa84bea6 to 0xa2afbee 2024-04-24T00:35:09.446 INFO:teuthology.orchestra.run.smithi064.stdout:2481 read 0x2c50eb3 thru 0x2c59b79 (0x8cc7 bytes) 2024-04-24T00:35:09.446 INFO:teuthology.orchestra.run.smithi064.stdout:2482 write 0x2b96e2a thru 0x2ba682d (0xfa04 bytes) 2024-04-24T00:35:09.457 INFO:teuthology.orchestra.run.smithi064.stdout:2483 read 0x39052ef thru 0x39062bc (0xfce bytes) 2024-04-24T00:35:09.457 INFO:teuthology.orchestra.run.smithi064.stdout:2484 clone 16 order 23 su 131072 sc 4 2024-04-24T00:35:09.485 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:09.479+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:09.506 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:09.506 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:09.506 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:09.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:09.586 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:09.582+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:09.687 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:09.683+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:09.692 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:09.692 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:09.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:09.692 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.978945887000002, 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.693 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.978847962000003, 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.698 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:09.704 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.619468430000001, 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:35:09.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:35:09.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30133 3.1f e66/59 ondisk, result = 0)", 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.606874883, 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060466079999999998, 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080440+0000", 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080446+0000", 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6699999999999999e-06 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080465+0000", 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080476+0000", 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0937999999999999e-05 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140869+0000", 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060392759999999997 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140882+0000", 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3131e-05 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140905+0000", 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2642000000000001e-05 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.713 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30705 3.2 3:45c09e0c:::rbd_data.115e8210e51a.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.434861969, 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063528001000000001, 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30705 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252455+0000", 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.303e-06 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252477+0000", 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1970000000000001e-05 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252515+0000", 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7616e-05 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252536+0000", 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376999999999999e-05 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252696+0000", 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160192 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252945+0000", 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024906400000000001 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.256753+0000", 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038072129999999998 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315658+0000", 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058905258000000002 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315790+0000", 2024-04-24T00:35:09.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013223999999999999 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315894+0000", 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103886 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315980+0000", 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5882000000000002e-05 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30705 3.2 e66/53 ondisk, result = 0)", 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.419649866, 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048330598000000002, 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267667+0000", 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.96e-06 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267683+0000", 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6336000000000001e-05 2024-04-24T00:35:09.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267730+0000", 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7079000000000001e-05 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267757+0000", 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6364e-05 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315724+0000", 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047967018 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315742+0000", 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7935999999999999e-05 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315995+0000", 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025290499999999999 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.723 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33506 3.c 3:3787ff34:::rbd_data.115ec4f9b3a7.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.889265824999999, 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061037313000000003, 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33506 2024-04-24T00:35:09.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798052+0000", 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798067+0000", 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5160000000000001e-05 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798107+0000", 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9946e-05 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798130+0000", 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2637e-05 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798238+0000", 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010841299999999999 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798344+0000", 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010631199999999999 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.801407+0000", 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030623600000000001 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858616+0000", 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057209342000000003 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858879+0000", 2024-04-24T00:35:09.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000262952 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858992+0000", 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113429 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859085+0000", 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2925999999999998e-05 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33506 3.c e70/56 ondisk, result = 0)", 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.879633058, 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051419185999999999, 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:09.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807682+0000", 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807687+0000", 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807704+0000", 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7424000000000001e-05 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807714+0000", 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3400000000000004e-06 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858714+0000", 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051000248999999998 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858732+0000", 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7995999999999998e-05 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859100+0000", 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036787400000000002 2024-04-24T00:35:09.733 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35694 3.1b 3:dcf0eb61:::rbd_data.115ec5833ea9.00000000000000aa:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.767817207, 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049399446, 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.734 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35694 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919498+0000", 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919504+0000", 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919520+0000", 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5908000000000001e-05 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919532+0000", 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2964e-05 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919630+0000", 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7096999999999995e-05 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919734+0000", 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104128 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.922056+0000", 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023219149999999999 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968775+0000", 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046719274999999998 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968817+0000", 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-05 2024-04-24T00:35:09.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968858+0000", 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1440999999999998e-05 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968896+0000", 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7741000000000003e-05 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.796484443000001, 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.740 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:35:09.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44664 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 524288~131072 [fadvise_sequential+fadvise_nocache],write 1179648~1048576 [fadvise_sequential+fadvise_nocache],write 10616832~2048 [fadvise_sequential+fadvise_nocache],write 10627072~4096 [fadvise_sequential+fadvise_nocache],write 10639360~4096 [fadvise_sequential+fadvise_nocache],write 10651648~4096 [fadvise_sequential+fadvise_nocache],write 10663936~4096 [fadvise_sequential+fadvise_nocache],write 10676224~4096 [fadvise_sequential+fadvise_nocache],write 10688512~4096 [fadvise_sequential+fadvise_nocache],write 10700800~4096 [fadvise_sequential+fadvise_nocache],write 10713088~4096 [fadvise_sequential+fadvise_nocache],write 10725376~4096 [fadvise_sequential+fadvise_nocache],write 10737664~4096 [fadvise_sequential+fadvise_nocache],write 10754048~4096 [fadvise_sequential+fadvise_nocache],write 10766336~4096 [fadvise_sequential+fadvise_nocache],write 10778624~4096 [fadvise_sequential+fadvise_nocache],write 10790912~4096 [fadvise_sequential+fadvise_nocache],write 10803200~4096 [fadvise_sequential+fadvise_nocache],write 10815488~4096 [fadvise_sequential+fadvise_nocache],write 10827776~4096 [fadvise_sequential+fadvise_nocache],write 10840064~4096 [fadvise_sequential+fadvise_nocache],write 10852352~4096 [fadvise_sequential+fadvise_nocache],write 10864640~4096 [fadvise_sequential+fadvise_nocache],write 10876928~2048 [fadvise_sequential+fadvise_nocache],write 10881024~4096 [fadvise_sequential+fadvise_nocache],write 10893312~4096 [fadvise_sequential+fadvise_nocache],write 10905600~4096 [fadvise_sequential+fadvise_nocache],write 10917888~4096 [fadvise_sequential+fadvise_nocache],write 10930176~4096 [fadvise_sequential+fadvise_nocache],write 10942464~4096 [fadvise_sequential+fadvise_nocache],write 10954752~4096 [fadvise_sequential+fadvise_nocache],write 10967040~4096 [fadvise_sequential+fadvise_nocache],write 10979328~4096 [fadvise_sequential+fadvise_nocache],write 10991616~4096 [fadvise_sequential+fadvise_nocache],write 11003904~4096 [fadvise_sequential+fadvise_nocache],write 11010048~2048 [fadvise_sequential+fadvise_nocache],write 11020288~4096 [fadvise_sequential+fadvise_nocache],write 11032576~4096 [fadvise_sequential+fadvise_nocache],write 11044864~4096 [fadvise_sequential+fadvise_nocache],write 11057152~4096 [fadvise_sequential+fadvise_nocache],write 11069440~4096 [fadvise_sequential+fadvise_nocache],write 11081728~4096 [fadvise_sequential+fadvise_nocache],write 11094016~4096 [fadvise_sequential+fadvise_nocache],write 11106304~4096 [fadvise_sequential+fadvise_nocache],write 11118592~4096 [fadvise_sequential+fadvise_nocache],write 11130880~4096 [fadvise_sequential+fadvise_nocache],write 14417920~4096 [fadvise_sequential+fadvise_nocache],write 14426112~8192 [fadvise_sequential+fadvise_nocache],write 14438400~8192 [fadvise_sequential+fadvise_nocache],write 14450688~8192 [fadvise_sequential+fadvise_nocache],write 14462976~8192 [fadvise_sequential+fadvise_nocache],write 14475264~8192 [fadvise_sequential+fadvise_nocache],write 14487552~8192 [fadvise_sequential+fadvise_nocache],write 14499840~8192 [fadvise_sequential+fadvise_nocache],write 14512128~8192 [fadvise_sequential+fadvise_nocache],write 14524416~8192 [fadvise_sequential+fadvise_nocache],write 14536704~8192 [fadvise_sequential+fadvise_nocache],write 14553088~8192 [fadvise_sequential+fadvise_nocache],write 14565376~8192 [fadvise_sequential+fadvise_nocache],write 14577664~8192 [fadvise_sequential+fadvise_nocache],write 14589952~8192 [fadvise_sequential+fadvise_nocache],write 14602240~8192 [fadvise_sequential+fadvise_nocache],write 14614528~8192 [fadvise_sequential+fadvise_nocache],write 14626816~8192 [fadvise_sequential+fadvise_nocache],write 14639104~8192 [fadvise_sequential+fadvise_nocache],write 14651392~8192 [fadvise_sequential+fadvise_nocache],write 14663680~8192 [fadvise_sequential+fadvise_nocache],write 14675968~4096 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.784411122, 2024-04-24T00:35:09.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768734, 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44664 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902904+0000", 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5179999999999999e-06 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.904804+0000", 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018996390000000001 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905393+0000", 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058872600000000003 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905408+0000", 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5073e-05 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905482+0000", 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4284999999999998e-05 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905785+0000", 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030266100000000002 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.907249+0000", 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001464899 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.932396+0000", 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025146379 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958308+0000", 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025912418999999999 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958527+0000", 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218297 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958671+0000", 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144838 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.750 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46463 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:1d [read 360448~8192 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0892344450000007, 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042258337999999999, 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46463 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.751 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598080+0000", 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598083+0000", 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6740000000000001e-06 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598091+0000", 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1089999999999998e-06 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598097+0000", 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5370000000000003e-06 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.635329+0000", 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037231775000000002 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.635418+0000", 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8629999999999997e-05 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640338+0000", 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049199730000000002 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46464 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0892085219999998, 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042285080000000003, 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46464 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598106+0000", 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7700000000000003e-07 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598109+0000", 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8080000000000001e-06 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598117+0000", 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2430000000000005e-06 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598124+0000", 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6000000000000003e-06 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640379+0000", 2024-04-24T00:35:09.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042255297999999997 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640390+0000", 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1253999999999999e-05 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46489 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598717+0000", 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0885963709999995, 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041719732000000002, 2024-04-24T00:35:09.758 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46489 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598717+0000", 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598717+0000", 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598718+0000", 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4e-07 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598732+0000", 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3906e-05 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598749+0000", 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6908000000000001e-05 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598761+0000", 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2232000000000001e-05 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640427+0000", 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041665617000000002 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640437+0000", 2024-04-24T00:35:09.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0229e-05 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46490 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598775+0000", 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0885387170000005, 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041708658000000003, 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:09.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46490 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598775+0000", 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598775+0000", 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598776+0000", 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598781+0000", 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6689999999999997e-06 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598795+0000", 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4694e-05 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598807+0000", 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1765e-05 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640474+0000", 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041666630000000003 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640484+0000", 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0000000000000001e-05 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.765 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46491 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598820+0000", 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0884937749999999, 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041702850999999999, 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46491 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.766 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598820+0000", 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598820+0000", 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598821+0000", 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.026e-06 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598826+0000", 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4889999999999997e-06 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598840+0000", 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4528e-05 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598852+0000", 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2106000000000001e-05 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640516+0000", 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041664113000000003 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640523+0000", 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5889999999999999e-06 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46492 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598976+0000", 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0883376729999998, 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041582613999999997, 2024-04-24T00:35:09.769 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46492 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598976+0000", 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598976+0000", 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598978+0000", 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598986+0000", 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6909999999999997e-06 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599005+0000", 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9777000000000001e-05 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.599020+0000", 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5197e-05 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640552+0000", 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041532001999999998 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640559+0000", 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2099999999999998e-06 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.2087243040000004, 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:09.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:09.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:35:09.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:35:09.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:35:09.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.1834556980000004, 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:35:09.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:35:09.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:67305 3.17 e89/59 ondisk, result = 0)", 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.83268162199999995, 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049974565999999998, 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:09.782 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854633+0000", 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854637+0000", 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.946e-06 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854649+0000", 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2507000000000001e-05 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854658+0000", 2024-04-24T00:35:09.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6330000000000002e-06 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904468+0000", 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049809535000000002 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904479+0000", 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-05 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904607+0000", 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012738300000000001 2024-04-24T00:35:09.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.786 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:09.786 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:09.786 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:09.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:09.788 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:09.782+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:09.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:35:09.889 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:09.886+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:09.958 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:09.959 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:09.959 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:09.959 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:09.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:10.022 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:10.019+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:10.091 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:10.086+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:10.095 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:10.095 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:10.095 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:10.095 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:10.095 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:10.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:10.194 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:10.191+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10769 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.987914934000003, 2024-04-24T00:35:10.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012818598, 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10769 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:35:10.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282742+0000", 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282747+0000", 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0329999999999999e-06 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282759+0000", 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1687e-05 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282768+0000", 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.286885+0000", 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041171139999999998 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.287035+0000", 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149968 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.288104+0000", 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010690540000000001 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295331+0000", 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072273800000000003 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295418+0000", 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295493+0000", 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4825000000000003e-05 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295560+0000", 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7191000000000001e-05 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10770 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.987875684999999, 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013929258999999999, 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.282 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10770 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282782+0000", 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9000000000000005e-07 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282786+0000", 2024-04-24T00:35:10.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1330000000000001e-06 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282795+0000", 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-06 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282803+0000", 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8499999999999994e-06 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289625+0000", 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068213689999999999 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289770+0000", 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014539899999999999 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.290476+0000", 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070591399999999998 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295352+0000", 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048766089999999996 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296546+0000", 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011937580000000001 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296638+0000", 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2179e-05 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296710+0000", 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1527999999999997e-05 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.987842585000003, 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.288 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:35:10.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.987806571999997, 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.294 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:35:10.299 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.738402178000001, 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:10.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:35:10.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.704365125000002, 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.305 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.113639160999998, 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.309 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.314 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.104383036000002, 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:10.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.318 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30578 3.10 3:089a659d:::rbd_object_map.115e8210e51a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.351033505, 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012659043, 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.319 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30578 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919624+0000", 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919632+0000", 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5969999999999992e-06 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919642+0000", 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8230000000000006e-06 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919650+0000", 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6529999999999994e-06 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.927798+0000", 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0081486619999999992 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.928005+0000", 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020605499999999999 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.929327+0000", 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013227689999999999 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931324+0000", 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019969150000000001 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931986+0000", 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066179100000000001 2024-04-24T00:35:10.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.932159+0000", 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017306600000000001 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.932282+0000", 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012284199999999999 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.324 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.589251129000001, 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:35:10.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:35:10.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.561801167999999, 2024-04-24T00:35:10.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:35:10.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:42868 3.1e 3:781b6cfc:::rbd_data.115e56ad4d37.000000000000003c:snapdir [list-snaps] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:10.334 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.517099963, 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021882233000000001, 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 42868 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753558+0000", 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6869999999999999e-06 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753573+0000", 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.446e-05 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753615+0000", 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.227e-05 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753633+0000", 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8240000000000002e-05 2024-04-24T00:35:10.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.754227+0000", 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059412099999999997 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.775439+0000", 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021211455000000001 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44350 3.a 3:54cb0ce7:::rbd_data.115e56ad4d37.0000000000000002:1b [read 305152~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864718+0000", 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.40593863, 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012190103000000001, 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44350 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864718+0000", 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864718+0000", 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864718+0000", 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9100000000000004e-07 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864721+0000", 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7819999999999998e-06 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864728+0000", 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4420000000000003e-06 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864733+0000", 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0900000000000004e-06 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876458+0000", 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011725541000000001 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876560+0000", 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102032 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876908+0000", 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034762500000000001 2024-04-24T00:35:10.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44380 3.10 3:09b53198:::rbd_data.115e56ad4d37.000000000000000e:1b [read 458752~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864835+0000", 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.405821315000001, 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012604941999999999, 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.343 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44380 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864835+0000", 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864835+0000", 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864836+0000", 2024-04-24T00:35:10.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864838+0000", 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5569999999999998e-06 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864844+0000", 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9950000000000002e-06 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864849+0000", 2024-04-24T00:35:10.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8369999999999996e-06 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876972+0000", 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012123055000000001 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877059+0000", 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6625999999999994e-05 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877440+0000", 2024-04-24T00:35:10.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038138499999999998 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44414 3.a 3:54cbcdc1:::rbd_data.115e56ad4d37.0000000000000069:1b [read 22528~20480 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864855+0000", 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.405801359, 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012327214, 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.347 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44414 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864855+0000", 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864855+0000", 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864855+0000", 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.46e-07 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864864+0000", 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6950000000000006e-06 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864870+0000", 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9510000000000003e-06 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864875+0000", 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7779999999999998e-06 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876950+0000", 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012074830999999999 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877040+0000", 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0750999999999999e-05 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877182+0000", 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000141762 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44415 3.a 3:54cbcdc1:::rbd_data.115e56ad4d37.0000000000000069:1b [read 131072~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864881+0000", 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.405775152, 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012561035999999999, 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44415 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864881+0000", 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864881+0000", 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864882+0000", 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864890+0000", 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9910000000000002e-06 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864896+0000", 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2020000000000001e-06 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864901+0000", 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9559999999999996e-06 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877219+0000", 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012318223 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877335+0000", 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115913 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877442+0000", 2024-04-24T00:35:10.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107277 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44416 3.a 3:54cbcdc1:::rbd_data.115e56ad4d37.0000000000000069:1b [read 239616~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864908+0000", 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.405748581999999, 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012734291999999999, 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.355 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44416 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864908+0000", 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864908+0000", 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864908+0000", 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2599999999999998e-07 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864910+0000", 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0379999999999998e-06 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864916+0000", 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7799999999999997e-06 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864921+0000", 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.887e-06 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877466+0000", 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012544726000000001 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877540+0000", 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4565000000000002e-05 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877642+0000", 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010187 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44417 3.a 3:54cbcdc1:::rbd_data.115e56ad4d37.0000000000000069:1b [read 350208~20480 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864927+0000", 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.405729216999999, 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012946459, 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.359 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44417 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864927+0000", 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864927+0000", 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864928+0000", 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9999999999999998e-07 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864930+0000", 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864935+0000", 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6910000000000002e-06 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864940+0000", 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877678+0000", 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012738311 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877762+0000", 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3997000000000002e-05 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877874+0000", 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111197 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:66044 3.a 3:56b96bc9:::rbd_object_map.115e8561432:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-24T00:35:10.363 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.788069+0000", 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.4825874040000002, 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015982158999999999, 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66044 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788069+0000", 2024-04-24T00:35:10.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788069+0000", 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788075+0000", 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4620000000000001e-06 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788088+0000", 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2917e-05 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788118+0000", 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9473999999999999e-05 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788135+0000", 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7382999999999999e-05 2024-04-24T00:35:10.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788205+0000", 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0029000000000003e-05 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788284+0000", 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8683999999999994e-05 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788585+0000", 2024-04-24T00:35:10.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030155900000000002 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.790702+0000", 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021170009999999999 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.803778+0000", 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013075501999999999 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.803960+0000", 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018172500000000001 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.804051+0000", 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1422999999999995e-05 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:66045 3.a 3:56b96bc9:::rbd_object_map.115e8561432:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.788169+0000", 2024-04-24T00:35:10.369 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.4824868909999998, 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016081891000000001, 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66045 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788169+0000", 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788169+0000", 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788171+0000", 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.477e-06 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788178+0000", 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3339999999999996e-06 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788198+0000", 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9553999999999999e-05 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788212+0000", 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.45e-05 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.789171+0000", 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00095885499999999999 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.789328+0000", 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015645899999999999 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.789918+0000", 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059015999999999995 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.791277+0000", 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001358688 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.804166+0000", 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012889929 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.804213+0000", 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6213e-05 2024-04-24T00:35:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.804251+0000", 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8722e-05 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.375 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:10.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:10.377 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:10.371+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:10.477 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:10.474+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:10.545 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:10.545 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:10.545 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:10.545 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:35:10.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:10.607 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:10.603+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:10.679 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:10.675+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:10.694 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:10.694 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:10.695 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:10.695 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:10.695 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:10.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:10.794 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:10.791+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:10.876 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:35:10.876 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.944632964999997, 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:35:10.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:35:10.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:35:10.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.012667276999998, 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.883 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:10.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:35:10.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:35:10.888 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.011418528999997, 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.889 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:35:10.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.894 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.853794424999997, 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.840299881999997, 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.900 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:35:10.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.578518766999998, 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.904 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:10.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.324932623999999, 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.910 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:35:10.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:35:10.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:35:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:10.915 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.297258759999998, 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:10.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:10.920 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.80323594, 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:10.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:35:10.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:35:10.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:35:10.925 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.794227106000001, 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.292528158, 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:35:10.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:35:10.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.292465812, 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.934 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:35:10.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.938 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.291956132999999, 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:35:10.942 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.291845071000001, 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.943 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:35:10.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.291779805000001, 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:35:10.947 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:35:10.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.291719738999999, 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.955 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.291663046, 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.956 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:10.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:35:10.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:35:10.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.291593712999999, 2024-04-24T00:35:10.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:35:10.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.291503093999999, 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.965 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:35:10.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.291459680999999, 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:35:10.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:35:10.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:10.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:10.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:35:10.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:35:10.974 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.974 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.974 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.974 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:10.974 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:10.974 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:10.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:10.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:10.977 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:10.970+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:11.077 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:11.071+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:11.108 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:11.108 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:11.108 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:11.109 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:11.111 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:35:11.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:11.130 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:11.130 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:11.148 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:11.148 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:11.165 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:11.165 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:11.177 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:11.171+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:11.185 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:11.186 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:11.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:11.229 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:11.229 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:11.242 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:11.242 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:11.250 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:11.250 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:35:11.254 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:35:11.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:11.278 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:11.275+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:11.379 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:11.374+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:11.425 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:11.426 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:11.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:11.481 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:11.478+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:11.580 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:11.574+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:11.613 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:11.613 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:11.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:11.682 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:11.679+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:11.711 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-04-24T00:35:11.784 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:11.779+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:11.801 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:11.802 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:11.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:11.822 INFO:teuthology.orchestra.run.smithi064.stdout:2486 write 0x797d0de thru 0x798691a (0x983d bytes) 2024-04-24T00:35:11.854 INFO:teuthology.orchestra.run.smithi064.stdout:2488 write 0xbb1b396 thru 0xbb1b8f2 (0x55d bytes) 2024-04-24T00:35:11.860 INFO:teuthology.orchestra.run.smithi064.stdout:2491 writesame 0x81b9363 thru 0x81c078a data_size 0x3f(0x7428 bytes) 2024-04-24T00:35:11.869 INFO:teuthology.orchestra.run.smithi064.stdout:2493 writesame 0x6e6b92f thru 0x6e6ec85 data_size 0x1(0x3357 bytes) 2024-04-24T00:35:11.884 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:11.878+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:11.917 INFO:teuthology.orchestra.run.smithi064.stdout:2494 read 0xadc73ee thru 0xadd0abf (0x96d2 bytes) 2024-04-24T00:35:11.917 INFO:teuthology.orchestra.run.smithi064.stdout:2496 read 0xa8f2d16 thru 0xa902aba (0xfda5 bytes) 2024-04-24T00:35:11.919 INFO:teuthology.orchestra.run.smithi064.stdout:2498 write 0x87fb968 thru 0x88094d8 (0xdb71 bytes) 2024-04-24T00:35:11.932 INFO:teuthology.orchestra.run.smithi064.stdout:2499 writesame 0xb216eaa thru 0xb22052f data_size 0x1(0x9686 bytes) 2024-04-24T00:35:11.949 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:11.949 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:11.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:12.009 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:12.006+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:12.059 INFO:teuthology.orchestra.run.smithi064.stdout:2500 writesame 0x594e3ef thru 0x59558e3 data_size 0x1(0x74f5 bytes) 2024-04-24T00:35:12.085 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:12.082+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:12.126 INFO:teuthology.orchestra.run.smithi064.stdout:2501 read 0x72c7388 thru 0x72d6bef (0xf868 bytes) 2024-04-24T00:35:12.126 INFO:teuthology.orchestra.run.smithi064.stdout:2502 writesame 0x43ea8ec thru 0x43f0c41 data_size 0x5(0x6356 bytes) 2024-04-24T00:35:12.136 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:12.136 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:12.136 INFO:teuthology.orchestra.run.smithi064.stdout:2503 punch from 0xb45e6ef to 0xb46ca0e, (0xe31f bytes) 2024-04-24T00:35:12.137 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:12.137 INFO:teuthology.orchestra.run.smithi064.stdout:2506 read 0x1bbb651 thru 0x1bc09d8 (0x5388 bytes) 2024-04-24T00:35:12.138 INFO:teuthology.orchestra.run.smithi064.stdout:2508 punch from 0x585f9bc to 0x58649a9, (0x4fed bytes) 2024-04-24T00:35:12.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:12.154 INFO:teuthology.orchestra.run.smithi064.stdout:2509 writesame 0xcc59431 thru 0xcc5d70e data_size 0x1b(0x42de bytes) 2024-04-24T00:35:12.154 INFO:teuthology.orchestra.run.smithi064.stdout:2510 clone 17 order 18 su 65536 sc 4 2024-04-24T00:35:12.187 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:12.183+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:12.280 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:12.280 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:12.288 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:12.283+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:12.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:12.388 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:12.383+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:12.415 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:12.415 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:12.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:12.490 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:12.486+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:12.563 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:12.563 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:12.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:12.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:35:12.621 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:12.618+0000 7f13038ff7c0 -1 osd.0 74 log_to_monitors true 2024-04-24T00:35:12.622 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:12.618+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:12.690 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:12.686+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:12.706 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:12.706 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:12.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:12.792 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:12.787+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:12.844 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:12.845 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:12.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:12.893 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:12.886+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:12.993 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:12.987+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:13.023 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:13.023 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:13.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:13.126 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:13.123+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:13.195 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:13.191+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:13.205 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:13.205 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:13.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:13.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:13.302 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:13.298+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:13.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:13.387 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:13.387 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:13.394 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-24T00:35:13.396 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:13.390+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:13.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-04-24T00:35:13.450 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-04-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-04-24T00:35:13.452 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T00:35:13.453 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-24T00:35:13.498 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:13.494+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:13.522 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:13.523 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:13.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:13.574 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:35:13.598 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:13.594+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:13.605 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:13.602+0000 7f12f76dd640 -1 osd.0 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T00:35:13.633 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:13.661 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:13.662 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:13.685 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:13.685 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:13.689 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:13.689 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:13.699 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:13.695+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:13.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:13.710 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:13.710 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:13.733 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:13.733 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:13.756 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:13.756 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:13.780 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T00:35:13.785 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T00:35:13.825 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:13.822+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:13.902 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:13.895+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10769 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 59.632395827000003, 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012818598, 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.919 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10769 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282741+0000", 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282742+0000", 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4300000000000001e-07 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282747+0000", 2024-04-24T00:35:13.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0329999999999999e-06 2024-04-24T00:35:13.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282759+0000", 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1687e-05 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282768+0000", 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6559999999999998e-06 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.286885+0000", 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041171139999999998 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.287035+0000", 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149968 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.288104+0000", 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010690540000000001 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295331+0000", 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072273800000000003 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295418+0000", 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295493+0000", 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4825000000000003e-05 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295560+0000", 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7191000000000001e-05 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10770 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 59.632356578, 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013929258999999999, 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10770 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282781+0000", 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282782+0000", 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9000000000000005e-07 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282786+0000", 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1330000000000001e-06 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282795+0000", 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5300000000000002e-06 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282803+0000", 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8499999999999994e-06 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289625+0000", 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068213689999999999 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.289770+0000", 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014539899999999999 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.290476+0000", 2024-04-24T00:35:13.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070591399999999998 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.295352+0000", 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048766089999999996 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296546+0000", 2024-04-24T00:35:13.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011937580000000001 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296638+0000", 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2179e-05 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.296710+0000", 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1527999999999997e-05 2024-04-24T00:35:13.931 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.932 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.932 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.932 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:13.932 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:13.932 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 59.632323478000004, 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.933 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:35:13.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 59.632287464999997, 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.939 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:35:13.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:35:13.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.944 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.382883071000002, 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.945 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:13.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:13.950 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.348846018000003, 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:35:13.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.758120054000003, 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.748863929000002, 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:35:13.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:35:13.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30578 3.10 3:089a659d:::rbd_object_map.115e8210e51a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.995514398000001, 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012659043, 2024-04-24T00:35:13.964 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30578 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919623+0000", 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919624+0000", 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919632+0000", 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5969999999999992e-06 2024-04-24T00:35:13.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919642+0000", 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8230000000000006e-06 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.919650+0000", 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6529999999999994e-06 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.927798+0000", 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0081486619999999992 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.928005+0000", 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020605499999999999 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.929327+0000", 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013227689999999999 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931324+0000", 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019969150000000001 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.931986+0000", 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066179100000000001 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.932159+0000", 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017306600000000001 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.932282+0000", 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012284199999999999 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.233732022000002, 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.970 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:35:13.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:35:13.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.206282061, 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:35:13.976 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:13.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.980 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:42868 3.1e 3:781b6cfc:::rbd_data.115e56ad4d37.000000000000003c:snapdir [list-snaps] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.161580856, 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021882233000000001, 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.981 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 42868 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753558+0000", 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6869999999999999e-06 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753573+0000", 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.446e-05 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753615+0000", 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.227e-05 2024-04-24T00:35:13.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753633+0000", 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8240000000000002e-05 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.754227+0000", 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059412099999999997 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.775439+0000", 2024-04-24T00:35:13.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021211455000000001 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44350 3.a 3:54cb0ce7:::rbd_data.115e56ad4d37.0000000000000002:1b [read 305152~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864718+0000", 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.050419522999999, 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012190103000000001, 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.985 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44350 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864718+0000", 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864718+0000", 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864718+0000", 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9100000000000004e-07 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864721+0000", 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7819999999999998e-06 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864728+0000", 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4420000000000003e-06 2024-04-24T00:35:13.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864733+0000", 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0900000000000004e-06 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876458+0000", 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011725541000000001 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876560+0000", 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102032 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876908+0000", 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034762500000000001 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.989 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44380 3.10 3:09b53198:::rbd_data.115e56ad4d37.000000000000000e:1b [read 458752~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864835+0000", 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.050302208000002, 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012604941999999999, 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.990 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44380 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864835+0000", 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864835+0000", 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864836+0000", 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8699999999999995e-07 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864838+0000", 2024-04-24T00:35:13.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5569999999999998e-06 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864844+0000", 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9950000000000002e-06 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864849+0000", 2024-04-24T00:35:13.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8369999999999996e-06 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876972+0000", 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012123055000000001 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877059+0000", 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6625999999999994e-05 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877440+0000", 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038138499999999998 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44414 3.a 3:54cbcdc1:::rbd_data.115e56ad4d37.0000000000000069:1b [read 22528~20480 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864855+0000", 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.050282251999999, 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012327214, 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:13.995 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44414 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864855+0000", 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864855+0000", 2024-04-24T00:35:13.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864855+0000", 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.46e-07 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864864+0000", 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6950000000000006e-06 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:13.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864870+0000", 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9510000000000003e-06 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864875+0000", 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7779999999999998e-06 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.876950+0000", 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012074830999999999 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877040+0000", 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0750999999999999e-05 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877182+0000", 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000141762 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44415 3.a 3:54cbcdc1:::rbd_data.115e56ad4d37.0000000000000069:1b [read 131072~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864881+0000", 2024-04-24T00:35:13.999 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.050256045000001, 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012561035999999999, 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44415 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864881+0000", 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864881+0000", 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864882+0000", 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864890+0000", 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9910000000000002e-06 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864896+0000", 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2020000000000001e-06 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864901+0000", 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9559999999999996e-06 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877219+0000", 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012318223 2024-04-24T00:35:14.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877335+0000", 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115913 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877442+0000", 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107277 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.003 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44416 3.a 3:54cbcdc1:::rbd_data.115e56ad4d37.0000000000000069:1b [read 239616~22528 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864908+0000", 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.050229474999998, 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012734291999999999, 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44416 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864908+0000", 2024-04-24T00:35:14.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864908+0000", 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864908+0000", 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2599999999999998e-07 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864910+0000", 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0379999999999998e-06 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864916+0000", 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7799999999999997e-06 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864921+0000", 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.887e-06 2024-04-24T00:35:14.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877466+0000", 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012544726000000001 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877540+0000", 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4565000000000002e-05 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877642+0000", 2024-04-24T00:35:14.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010187 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44417 3.a 3:54cbcdc1:::rbd_data.115e56ad4d37.0000000000000069:1b [read 350208~20480 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.864927+0000", 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.050210109999998, 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012946459, 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.008 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44417 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864927+0000", 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864927+0000", 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864928+0000", 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9999999999999998e-07 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864930+0000", 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864935+0000", 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6910000000000002e-06 2024-04-24T00:35:14.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.864940+0000", 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877678+0000", 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012738311 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877762+0000", 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3997000000000002e-05 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.877874+0000", 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111197 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:66044 3.a 3:56b96bc9:::rbd_object_map.115e8561432:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.788069+0000", 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.1270682969999992, 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015982158999999999, 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66044 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788069+0000", 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788069+0000", 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788075+0000", 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4620000000000001e-06 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788088+0000", 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2917e-05 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788118+0000", 2024-04-24T00:35:14.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9473999999999999e-05 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788135+0000", 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7382999999999999e-05 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788205+0000", 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0029000000000003e-05 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788284+0000", 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8683999999999994e-05 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788585+0000", 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030155900000000002 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.790702+0000", 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021170009999999999 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.803778+0000", 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013075501999999999 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.803960+0000", 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018172500000000001 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.804051+0000", 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1422999999999995e-05 2024-04-24T00:35:14.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:66045 3.a 3:56b96bc9:::rbd_object_map.115e8561432:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.788169+0000", 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.1269677839999996, 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016081891000000001, 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 66045 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788169+0000", 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788169+0000", 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788171+0000", 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.477e-06 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788178+0000", 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3339999999999996e-06 2024-04-24T00:35:14.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788198+0000", 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9553999999999999e-05 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.788212+0000", 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.45e-05 2024-04-24T00:35:14.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.789171+0000", 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00095885499999999999 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.789328+0000", 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015645899999999999 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.789918+0000", 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059015999999999995 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.791277+0000", 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001358688 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.804166+0000", 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012889929 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.804213+0000", 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6213e-05 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.804251+0000", 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8722e-05 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.024 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.025 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.025 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:14.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:14.027 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:14.019+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:14.127 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:14.122+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:14.176 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:14.176 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:14.176 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:14.176 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:14.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:14.228 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:14.222+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:14.307 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:14.307 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:14.307 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:14.307 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:14.308 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:14.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:14.366 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:14.363+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:14.430 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:14.426+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:14.443 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 93.514397368999994, 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.444 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:35:14.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:35:14.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.450 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.582431681000003, 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.451 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:35:14.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.581182933000001, 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.457 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:14.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:14.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:14.462 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.423558829000001, 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:14.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:35:14.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:35:14.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.410064286000001, 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.468 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:35:14.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.148283171000003, 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.894697028000003, 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.867023164000003, 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:14.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.373000343999998, 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:35:14.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:35:14.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:35:14.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.363991509999998, 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:35:14.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.862292562, 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:35:14.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.862230216, 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.503 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:14.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:35:14.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.861720537, 2024-04-24T00:35:14.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.861609475, 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:35:14.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.861544209, 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.516 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:14.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:35:14.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.861484143, 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:35:14.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:35:14.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:35:14.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.861427450000001, 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:14.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:35:14.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:35:14.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.861358117, 2024-04-24T00:35:14.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:14.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.533 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.861267498, 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:35:14.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:35:14.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:35:14.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:35:14.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.861224085, 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:14.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:14.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:35:14.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:14.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:14.542 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:14.544 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:14.539+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:14.544 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:14.645 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:14.639+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:14.691 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:14.691 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:14.691 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:14.691 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:14.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:14.745 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:14.739+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:14.783 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-04-24T00:35:14.845 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:14.839+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:14.847 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:14.847 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:14.847 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:14.847 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:14.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:14.947 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:14.943+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.557312568, 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:35:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:35:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:35:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.501801082, 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:35:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.401224434, 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:35:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.359659480000001, 2024-04-24T00:35:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44651 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head [write 0~524288 [fadvise_sequential+fadvise_nocache],write 655360~131072 [fadvise_sequential+fadvise_nocache],write 1310720~1048576 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~4096 [fadvise_sequential+fadvise_nocache],write 11155456~4096 [fadvise_sequential+fadvise_nocache],write 11167744~4096 [fadvise_sequential+fadvise_nocache],write 11180032~4096 [fadvise_sequential+fadvise_nocache],write 11192320~4096 [fadvise_sequential+fadvise_nocache],write 11204608~4096 [fadvise_sequential+fadvise_nocache],write 11216896~4096 [fadvise_sequential+fadvise_nocache],write 11229184~4096 [fadvise_sequential+fadvise_nocache],write 11241472~4096 [fadvise_sequential+fadvise_nocache],write 11253760~4096 [fadvise_sequential+fadvise_nocache],write 11266048~4096 [fadvise_sequential+fadvise_nocache],write 14548992~8192 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~12288 [fadvise_sequential+fadvise_nocache],write 14688256~8192 [fadvise_sequential+fadvise_nocache],write 14700544~8192 [fadvise_sequential+fadvise_nocache],write 14712832~8192 [fadvise_sequential+fadvise_nocache],write 14725120~8192 [fadvise_sequential+fadvise_nocache],write 14737408~8192 [fadvise_sequential+fadvise_nocache],write 14749696~8192 [fadvise_sequential+fadvise_nocache],write 14761984~8192 [fadvise_sequential+fadvise_nocache],write 14774272~8192 [fadvise_sequential+fadvise_nocache],write 14786560~8192 [fadvise_sequential+fadvise_nocache],write 14798848~6453 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:35:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.106999614999999, 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058685117000000002, 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44651 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:35:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884307+0000", 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0850000000000001e-06 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886333+0000", 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020258310000000001 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886370+0000", 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6485000000000001e-05 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886385+0000", 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5897e-05 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886564+0000", 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178455 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.887161+0000", 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059714499999999999 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.888890+0000", 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001729349 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920023+0000", 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031132238999999999 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942736+0000", 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02271306 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942884+0000", 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014812199999999999 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942989+0000", 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105449 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46434 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578656+0000", 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412647329, 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026471701, 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46434 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578656+0000", 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578656+0000", 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578657+0000", 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5000000000000001e-07 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578663+0000", 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5500000000000002e-06 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578684+0000", 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1169e-05 2024-04-24T00:35:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578702+0000", 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7390000000000001e-05 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.604458+0000", 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025756366999999999 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.604519+0000", 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0696999999999998e-05 2024-04-24T00:35:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.605128+0000", 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060957800000000005 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46435 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578719+0000", 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412584653, 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027308378000000001, 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46435 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578719+0000", 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578719+0000", 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578720+0000", 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578725+0000", 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9559999999999996e-06 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578744+0000", 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9173000000000001e-05 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578759+0000", 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4596999999999999e-05 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.605215+0000", 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026456218 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.605342+0000", 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012719700000000001 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606028+0000", 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068533699999999995 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46436 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578777+0000", 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412527251, 2024-04-24T00:35:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028104427000000001, 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46436 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578777+0000", 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578777+0000", 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578778+0000", 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578783+0000", 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9699999999999998e-06 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578801+0000", 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8525999999999998e-05 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578819+0000", 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7569999999999999e-05 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606119+0000", 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027300843000000002 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606247+0000", 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012771700000000001 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606881+0000", 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063380800000000003 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46437 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578834+0000", 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412470036, 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029803969999999999, 2024-04-24T00:35:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46437 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578834+0000", 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578834+0000", 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578835+0000", 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3399999999999997e-07 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578840+0000", 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8760000000000004e-06 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578858+0000", 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.861e-05 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578870+0000", 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2217e-05 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606974+0000", 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028103633999999999 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.607198+0000", 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000224019 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.608638+0000", 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00143968 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46438 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578888+0000", 2024-04-24T00:35:15.036 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412415633, 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031231299000000001, 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46438 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578888+0000", 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578888+0000", 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578889+0000", 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.16e-07 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578894+0000", 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8729999999999998e-06 2024-04-24T00:35:15.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578912+0000", 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8499999999999999e-05 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578925+0000", 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.225e-05 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.608693+0000", 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029768467 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.608780+0000", 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7012e-05 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.610119+0000", 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013392809999999999 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46439 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578942+0000", 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412361961, 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032570504, 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46439 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578942+0000", 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578942+0000", 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578943+0000", 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578947+0000", 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7199999999999997e-06 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578966+0000", 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:35:15.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578978+0000", 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2113000000000001e-05 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.610203+0000", 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031225148000000001 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.610368+0000", 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016536400000000001 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.611512+0000", 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011438590000000001 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46440 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578995+0000", 2024-04-24T00:35:15.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412308672, 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033872845999999998, 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46440 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578995+0000", 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578995+0000", 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578996+0000", 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5300000000000003e-07 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579001+0000", 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6700000000000002e-06 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579019+0000", 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8272999999999999e-05 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579031+0000", 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1744e-05 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.611603+0000", 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032572469999999999 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.611738+0000", 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013527299999999999 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.612868+0000", 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011295630000000001 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46465 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 401408~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579084+0000", 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412219873, 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035121767999999998, 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.050 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46465 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579084+0000", 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579084+0000", 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:35:15.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3659999999999999e-06 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579092+0000", 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.973e-06 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579113+0000", 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1123000000000001e-05 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579138+0000", 2024-04-24T00:35:15.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4807000000000002e-05 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.612958+0000", 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033820173000000002 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.613094+0000", 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135544 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.614206+0000", 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001111782 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46466 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 417792~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579162+0000", 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412142025, 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036326009999999999, 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46466 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579162+0000", 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579162+0000", 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579163+0000", 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.066e-06 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579169+0000", 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7400000000000001e-06 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579186+0000", 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7694e-05 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579201+0000", 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4646e-05 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.614296+0000", 2024-04-24T00:35:15.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035094822999999997 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.614430+0000", 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134661 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.615488+0000", 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010573799999999999 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46467 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 434176~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579220+0000", 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.412083996, 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036809401999999998, 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46467 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.059 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579220+0000", 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579220+0000", 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579221+0000", 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0529999999999999e-06 2024-04-24T00:35:15.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579226+0000", 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4029999999999998e-06 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579268+0000", 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1498999999999997e-05 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:35:15.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5104e-05 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.615569+0000", 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036286232000000002 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.615643+0000", 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3569e-05 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.616029+0000", 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038654200000000003 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46468 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 450560~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579352+0000", 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.411951405, 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037665068000000003, 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46468 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579352+0000", 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579352+0000", 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579353+0000", 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.33e-07 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0239999999999997e-06 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579375+0000", 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6432000000000001e-05 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579385+0000", 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.012e-05 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.616131+0000", 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036745932000000002 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.616288+0000", 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015730400000000001 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617017+0000", 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072932300000000004 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46469 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 466944~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579398+0000", 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.411906248999999, 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038520933, 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.067 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46469 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579398+0000", 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579398+0000", 2024-04-24T00:35:15.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579398+0000", 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6600000000000005e-07 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579402+0000", 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3739999999999998e-06 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579414+0000", 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1956e-05 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579421+0000", 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2660000000000004e-06 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617129+0000", 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037707683999999998 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617219+0000", 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0223000000000007e-05 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617918+0000", 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069956399999999996 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46470 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 483328~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.071 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579430+0000", 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.41187386, 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039293489000000001, 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46470 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579430+0000", 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579430+0000", 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579431+0000", 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9999999999999997e-07 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579434+0000", 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.146e-06 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579448+0000", 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4344e-05 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579461+0000", 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2836e-05 2024-04-24T00:35:15.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617968+0000", 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038507118999999999 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.618038+0000", 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9498000000000004e-05 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.618723+0000", 2024-04-24T00:35:15.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068584600000000005 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46471 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 499712~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579477+0000", 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.411826988, 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040936221000000002, 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46471 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579477+0000", 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579477+0000", 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579478+0000", 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8999999999999995e-07 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579481+0000", 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5470000000000001e-06 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579491+0000", 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7829999999999993e-06 2024-04-24T00:35:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579498+0000", 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3100000000000003e-06 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.618828+0000", 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03932973 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.618984+0000", 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015575800000000001 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.620413+0000", 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001429303 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46472 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 516096~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579513+0000", 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.411790527999999, 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041758379999999998, 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46472 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579513+0000", 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579513+0000", 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579514+0000", 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1999999999999998e-07 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579519+0000", 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9969999999999998e-06 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6002e-05 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4800000000000004e-06 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.620452+0000", 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040909214999999999 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.620562+0000", 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010995800000000001 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.621272+0000", 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070980800000000003 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.084 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:15.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:15.086 INFO:teuthology.orchestra.run.smithi064.stdout:2511 read 0x5eb680d thru 0x5ec60f0 (0xf8e4 bytes) 2024-04-24T00:35:15.086 INFO:teuthology.orchestra.run.smithi064.stdout:2513 writesame 0x8c1ef37 thru 0x8c2a42b data_size 0x11(0xb4f5 bytes) 2024-04-24T00:35:15.087 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:15.079+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:15.112 INFO:teuthology.orchestra.run.smithi064.stdout:2514 trunc from 0xcc5d70f to 0x3bef123 2024-04-24T00:35:15.186 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:15.181+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:15.241 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:15.241 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:15.245 INFO:teuthology.orchestra.run.smithi064.stdout:2515 punch from 0x22b9666 to 0x22bd633, (0x3fcd bytes) 2024-04-24T00:35:15.249 INFO:teuthology.orchestra.run.smithi064.stdout:2516 trunc from 0x3bef123 to 0x1905107 2024-04-24T00:35:15.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:15.297 INFO:teuthology.orchestra.run.smithi064.stdout:2517 trunc from 0x1905107 to 0xe521196 2024-04-24T00:35:15.297 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:15.289+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:15.315 INFO:teuthology.orchestra.run.smithi064.stdout:2518 trunc from 0xe521196 to 0x8c17526 2024-04-24T00:35:15.339 INFO:teuthology.orchestra.run.smithi064.stdout:2519 writesame 0xa93ee85 thru 0xa94728a data_size 0x2b(0x8406 bytes) 2024-04-24T00:35:15.339 INFO:teuthology.orchestra.run.smithi064.stdout:2520 read 0x68a368c thru 0x68a772c (0x40a1 bytes) 2024-04-24T00:35:15.340 INFO:teuthology.orchestra.run.smithi064.stdout:2521 write 0x6bb5c0c thru 0x6bbbec3 (0x62b8 bytes) 2024-04-24T00:35:15.347 INFO:teuthology.orchestra.run.smithi064.stdout:2522 clone 18 order 20 su 262144 sc 3 2024-04-24T00:35:15.388 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:15.383+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:15.417 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:15.417 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:15.417 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:15.417 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:15.417 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:15.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:15.488 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:15.483+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:15.589 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:15.583+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:15.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:15.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.955581847999994, 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:35:15.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:35:15.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:35:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20552 3.9 e49/21 3:923662ec:::rbd_data.115ee4742374.000000000000000a:head v 49'88, mlcod=48'86)", 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.626888248, 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062734230000000002, 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973502+0000", 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.04e-06 2024-04-24T00:35:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973677+0000", 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473200000000001 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973714+0000", 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7117999999999999e-05 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973734+0000", 2024-04-24T00:35:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0102000000000002e-05 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973879+0000", 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145654 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973946+0000", 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6476999999999994e-05 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979687+0000", 2024-04-24T00:35:15.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057411140000000003 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979773+0000", 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6186000000000005e-05 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.017306992000002, 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:15.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:35:15.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:35:15.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.017233136999998, 2024-04-24T00:35:15.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.320538353000003, 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.515006394, 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:68687 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'253, mlcod=94'249)", 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.934994951, 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005827128, 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665394+0000", 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7700000000000004e-07 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665402+0000", 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4030000000000007e-06 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665414+0000", 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1467999999999999e-05 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665419+0000", 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1780000000000002e-06 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665485+0000", 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5944000000000004e-05 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665507+0000", 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2359999999999999e-05 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.671176+0000", 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056689130000000003 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.671220+0000", 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4184999999999998e-05 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:69871 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'404, mlcod=94'396)", 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.7679176239999999, 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055042449999999996, 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832471+0000", 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9200000000000001e-07 2024-04-24T00:35:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832475+0000", 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.873e-06 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832482+0000", 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832486+0000", 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2470000000000002e-06 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832537+0000", 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1233000000000002e-05 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832563+0000", 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5269000000000002e-05 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.837951+0000", 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053882720000000004 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.837975+0000", 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3683e-05 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:69977 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'424, mlcod=94'419)", 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.7481615800000001, 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056478099999999996, 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852227+0000", 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852231+0000", 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1489999999999996e-06 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852240+0000", 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0010000000000007e-06 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852245+0000", 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.498e-06 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852283+0000", 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8195000000000003e-05 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852298+0000", 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4911e-05 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.857827+0000", 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055296210000000002 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.857874+0000", 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6897999999999997e-05 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70740 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'523, mlcod=94'518)", 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.635007978, 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061709410000000001, 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965381+0000", 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.094e-06 2024-04-24T00:35:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965389+0000", 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9580000000000007e-06 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965423+0000", 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3787e-05 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965443+0000", 2024-04-24T00:35:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0390999999999998e-05 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965583+0000", 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013995500000000001 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965606+0000", 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3003999999999999e-05 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971482+0000", 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058757649999999998 2024-04-24T00:35:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971551+0000", 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8986999999999995e-05 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70743 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'524, mlcod=94'518)", 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.6343036500000001, 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055099820000000001, 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966086+0000", 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1969999999999999e-06 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966096+0000", 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.995e-06 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966113+0000", 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7538e-05 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966124+0000", 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966181+0000", 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6937000000000001e-05 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966199+0000", 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8002000000000001e-05 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971556+0000", 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053569029999999997 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971594+0000", 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8186999999999999e-05 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70750 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'527, mlcod=94'519)", 2024-04-24T00:35:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.632027079, 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005844307, 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968362+0000", 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4900000000000004e-07 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968371+0000", 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.619e-06 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968388+0000", 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7113999999999999e-05 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968401+0000", 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3495e-05 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968434+0000", 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3306000000000003e-05 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968446+0000", 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1668000000000001e-05 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.974121+0000", 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056745290000000002 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.974205+0000", 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4627000000000001e-05 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70759 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'530, mlcod=94'521)", 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.629316236, 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054257760000000002, 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971073+0000", 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4300000000000002e-07 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971078+0000", 2024-04-24T00:35:15.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971086+0000", 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4400000000000005e-06 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971091+0000", 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.763e-06 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971138+0000", 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6897000000000002e-05 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971150+0000", 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2480000000000001e-05 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.976394+0000", 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052434580000000003 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.976498+0000", 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103973 2024-04-24T00:35:15.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70804 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'539, mlcod=94'532)", 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.619728853, 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058417199999999999, 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980660+0000", 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.99e-07 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980665+0000", 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.95e-06 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980681+0000", 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6653000000000001e-05 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980686+0000", 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9420000000000003e-06 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980873+0000", 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018610100000000001 2024-04-24T00:35:15.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980889+0000", 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6884e-05 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986449+0000", 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055591700000000004 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986501+0000", 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2321000000000003e-05 2024-04-24T00:35:15.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70823 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'542, mlcod=94'536)", 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.6170987590000001, 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057985090000000003, 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983290+0000", 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983296+0000", 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7969999999999997e-06 2024-04-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983304+0000", 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4519999999999997e-06 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983309+0000", 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6809999999999997e-06 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983353+0000", 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3844999999999998e-05 2024-04-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983365+0000", 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.235e-05 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.988997+0000", 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005632176 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.989088+0000", 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0587999999999994e-05 2024-04-24T00:35:15.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70843 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'546, mlcod=94'537)", 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.614361927, 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059306039999999999, 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986027+0000", 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2e-07 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986031+0000", 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986039+0000", 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1880000000000002e-06 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986043+0000", 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2339999999999996e-06 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986133+0000", 2024-04-24T00:35:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9783000000000004e-05 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986146+0000", 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2816000000000001e-05 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.991875+0000", 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057291429999999999 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.991957+0000", 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2027999999999995e-05 2024-04-24T00:35:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70942 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'565, mlcod=94'560)", 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.5958171989999999, 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005499884, 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004572+0000", 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.576e-06 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004583+0000", 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0383e-05 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004603+0000", 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9738999999999999e-05 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004612+0000", 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6919999999999997e-06 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004691+0000", 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8771999999999997e-05 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004711+0000", 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0018e-05 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009995+0000", 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052835520000000004 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.010071+0000", 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6151999999999999e-05 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70944 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'567, mlcod=94'561)", 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.5934626590000001, 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059596270000000003, 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.678 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006927+0000", 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006936+0000", 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6719999999999999e-06 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006960+0000", 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4249000000000001e-05 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006975+0000", 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4259e-05 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.007069+0000", 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4477999999999999e-05 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.007088+0000", 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8332999999999999e-05 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012838+0000", 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057502830000000001 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012885+0000", 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7206999999999997e-05 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70966 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'570, mlcod=94'563)", 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.5911404060000001, 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059298320000000003, 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009249+0000", 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.155e-06 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009257+0000", 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.892e-06 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009275+0000", 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8272000000000001e-05 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009288+0000", 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3107e-05 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009373+0000", 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4561999999999994e-05 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009407+0000", 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4100000000000002e-05 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.015084+0000", 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056774750000000004 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.015177+0000", 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3269000000000004e-05 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70979 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'574, mlcod=94'565)", 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.5881411010000002, 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054674650000000003, 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012248+0000", 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0920000000000001e-06 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012256+0000", 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9119999999999998e-06 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012273+0000", 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7280000000000001e-05 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012295+0000", 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1866999999999999e-05 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012370+0000", 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5278999999999995e-05 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012409+0000", 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8226999999999998e-05 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.017649+0000", 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052406350000000004 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:15.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.017714+0000", 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5172999999999994e-05 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:15.690 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:15.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:15.692 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:15.685+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:15.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:15.792 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:15.787+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:15.854 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:15.854 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:15.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:15.913 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:15.909+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:15.927 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:15.927 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-24T00:35:15.945 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:15.945 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:15.962 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:15.962 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:15.979 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:15.979 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:15.990 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:15.990 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:15.994 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:15.991+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:16.000 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:16.000 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:16.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:16.021 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:16.043 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:16.043 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:16.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:16.064 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:16.095 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:16.091+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:16.138 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:16.138 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:16.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:16.196 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:16.189+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:16.284 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:16.284 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:16.297 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:16.293+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:16.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:16.398 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:16.393+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:16.419 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:16.419 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:16.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:16.500 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:16.495+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:16.555 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:16.555 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:16.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:16.610 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:16.607+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:16.612 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:35:16.702 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:16.699+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:16.750 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:16.750 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:16.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:16.810 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:16.807+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:16.905 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:16.897+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:16.933 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:16.933 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:16.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:17.006 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:17.003+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:17.066 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:17.066 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:17.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:17.122 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:17.119+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:17.207 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:17.203+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:17.247 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:17.248 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:17.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:17.347 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:17.343+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:17.409 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:17.403+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:17.423 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:17.423 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:17.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:17.522 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:17.519+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:17.602 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:17.602 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:17.611 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:17.605+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:17.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:17.712 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:17.705+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:17.783 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:17.783 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:17.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:17.822 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-04-24T00:35:17.846 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:17.843+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:17.914 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:17.907+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:17.925 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:17.925 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:17.940 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:18.014 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:18.009+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:18.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:18.103 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:18.103 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:18.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:18.130 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:18.125+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:18.189 INFO:teuthology.orchestra.run.smithi064.stdout:2524 read 0x5a55b6b thru 0x5a5e114 (0x85aa bytes) 2024-04-24T00:35:18.191 INFO:teuthology.orchestra.run.smithi064.stdout:2525 trunc from 0xa94728b to 0xbf50f6d 2024-04-24T00:35:18.209 INFO:teuthology.orchestra.run.smithi064.stdout:2527 read 0xfb6764 thru 0xfbb1f0 (0x4a8d bytes) 2024-04-24T00:35:18.209 INFO:teuthology.orchestra.run.smithi064.stdout:2528 write 0x3023b36 thru 0x302a3a0 (0x686b bytes) 2024-04-24T00:35:18.216 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:18.211+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:18.218 INFO:teuthology.orchestra.run.smithi064.stdout:2529 punch from 0x8807777 to 0x8816a56, (0xf2df bytes) 2024-04-24T00:35:18.218 INFO:teuthology.orchestra.run.smithi064.stdout:2530 punch from 0x880d712 to 0x881cfc2, (0xf8b0 bytes) 2024-04-24T00:35:18.220 INFO:teuthology.orchestra.run.smithi064.stdout:2531 trunc from 0xbf50f6d to 0xa3c7704 2024-04-24T00:35:18.232 INFO:teuthology.orchestra.run.smithi064.stdout:2532 punch from 0x74df8f0 to 0x74e1d59, (0x2469 bytes) 2024-04-24T00:35:18.232 INFO:teuthology.orchestra.run.smithi064.stdout:2533 punch from 0x5b9ca28 to 0x5ba0e8e, (0x4466 bytes) 2024-04-24T00:35:18.233 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:18.233 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:18.243 INFO:teuthology.orchestra.run.smithi064.stdout:2538 trunc from 0xa3c7704 to 0x1f479b8 2024-04-24T00:35:18.253 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:18.253 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:18.258 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:18.258 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:18.259 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:18.259 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:18.267 INFO:teuthology.orchestra.run.smithi064.stdout:2540 read 0x15a02a8 thru 0x15aec8d (0xe9e6 bytes) 2024-04-24T00:35:18.267 INFO:teuthology.orchestra.run.smithi064.stdout:2541 read 0xdcaf10 thru 0xdcc74d (0x183e bytes) 2024-04-24T00:35:18.268 INFO:teuthology.orchestra.run.smithi064.stdout:2542 read 0xb5b37a thru 0xb5e0ab (0x2d32 bytes) 2024-04-24T00:35:18.268 INFO:teuthology.orchestra.run.smithi064.stdout:2544 read 0xd1e2ed thru 0xd2cb13 (0xe827 bytes) 2024-04-24T00:35:18.268 INFO:teuthology.orchestra.run.smithi064.stdout:2547 read 0x608c98 thru 0x60d9da (0x4d43 bytes) 2024-04-24T00:35:18.269 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:18.269 INFO:teuthology.orchestra.run.smithi064.stdout:2549 read 0x6d3076 thru 0x6d66c9 (0x3654 bytes) 2024-04-24T00:35:18.269 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:18.272 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:18.272 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:18.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:18.283 INFO:teuthology.orchestra.run.smithi064.stdout:2552 write 0x2e6ef2d thru 0x2e7ada4 (0xbe78 bytes) 2024-04-24T00:35:18.292 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:18.292 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:18.305 INFO:teuthology.orchestra.run.smithi064.stdout:2554 writesame 0x4fd62fb thru 0x4fd7835 data_size 0x1(0x153b bytes) 2024-04-24T00:35:18.306 INFO:teuthology.orchestra.run.smithi064.stdout:2556 read 0x133b8b9 thru 0x1342bba (0x7302 bytes) 2024-04-24T00:35:18.306 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:18.306 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:18.306 INFO:teuthology.orchestra.run.smithi064.stdout:2561 trunc from 0x4fd7836 to 0x4fe1e0d 2024-04-24T00:35:18.314 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:18.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:18.317 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:18.311+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:18.321 INFO:teuthology.orchestra.run.smithi064.stdout:2562 trunc from 0x4fe1e0d to 0x614cf7c 2024-04-24T00:35:18.335 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:18.335 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:18.340 INFO:teuthology.orchestra.run.smithi064.stdout:2563 trunc from 0x614cf7c to 0x4a4c304 2024-04-24T00:35:18.354 INFO:teuthology.orchestra.run.smithi064.stdout:2564 writesame 0x32a07f thru 0x332ff9 data_size 0x1(0x8f7b bytes) 2024-04-24T00:35:18.373 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:18.394 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:18.394 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:18.407 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:18.407 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:18.407 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:18.407 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:18.407 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:18.418 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:18.415+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:18.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:18.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:18.475 INFO:teuthology.orchestra.run.smithi064.stdout:2565 punch from 0x15c158a to 0x15c76f5, (0x616b bytes) 2024-04-24T00:35:18.475 INFO:teuthology.orchestra.run.smithi064.stdout:2566 punch from 0x38e610 to 0x392b3f, (0x452f bytes) 2024-04-24T00:35:18.475 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:18.499 INFO:teuthology.orchestra.run.smithi064.stdout:2569 writesame 0x89bb8ac thru 0x89c34dc data_size 0x1(0x7c31 bytes) 2024-04-24T00:35:18.514 INFO:teuthology.orchestra.run.smithi064.stdout:2570 write 0x8a4eccd thru 0x8a5bf59 (0xd28d bytes) 2024-04-24T00:35:18.518 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:18.513+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:18.523 INFO:teuthology.orchestra.run.smithi064.stdout:2571 read 0x25ee965 thru 0x25fd8fa (0xef96 bytes) 2024-04-24T00:35:18.523 INFO:teuthology.orchestra.run.smithi064.stdout:2572 read 0x1b806f2 thru 0x1b88c98 (0x85a7 bytes) 2024-04-24T00:35:18.523 INFO:teuthology.orchestra.run.smithi064.stdout:2573 punch from 0x6c0c88f to 0x6c1297a, (0x60eb bytes) 2024-04-24T00:35:18.523 INFO:teuthology.orchestra.run.smithi064.stdout:2574 read 0x3323262 thru 0x332f518 (0xc2b7 bytes) 2024-04-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi064.stdout:2576 trunc from 0x8a5bf5a to 0x53a2363 2024-04-24T00:35:18.554 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:18.554 INFO:teuthology.orchestra.run.smithi064.stdout:2581 read 0x16f5d31 thru 0x16fbce2 (0x5fb2 bytes) 2024-04-24T00:35:18.577 INFO:teuthology.orchestra.run.smithi064.stdout:2582 write 0x8f06186 thru 0x8f082ff (0x217a bytes) 2024-04-24T00:35:18.590 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:18.590 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:18.590 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:18.590 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:18.593 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:18.593 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.155471980999998, 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:18.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:35:18.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:35:18.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.099960494999998, 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:35:18.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:35:18.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:18.604 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.999383847000001, 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:35:18.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:35:18.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:18.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:35:18.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.957818892999999, 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:35:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:35:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:44651 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head [write 0~524288 [fadvise_sequential+fadvise_nocache],write 655360~131072 [fadvise_sequential+fadvise_nocache],write 1310720~1048576 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~4096 [fadvise_sequential+fadvise_nocache],write 11155456~4096 [fadvise_sequential+fadvise_nocache],write 11167744~4096 [fadvise_sequential+fadvise_nocache],write 11180032~4096 [fadvise_sequential+fadvise_nocache],write 11192320~4096 [fadvise_sequential+fadvise_nocache],write 11204608~4096 [fadvise_sequential+fadvise_nocache],write 11216896~4096 [fadvise_sequential+fadvise_nocache],write 11229184~4096 [fadvise_sequential+fadvise_nocache],write 11241472~4096 [fadvise_sequential+fadvise_nocache],write 11253760~4096 [fadvise_sequential+fadvise_nocache],write 11266048~4096 [fadvise_sequential+fadvise_nocache],write 14548992~8192 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~12288 [fadvise_sequential+fadvise_nocache],write 14688256~8192 [fadvise_sequential+fadvise_nocache],write 14700544~8192 [fadvise_sequential+fadvise_nocache],write 14712832~8192 [fadvise_sequential+fadvise_nocache],write 14725120~8192 [fadvise_sequential+fadvise_nocache],write 14737408~8192 [fadvise_sequential+fadvise_nocache],write 14749696~8192 [fadvise_sequential+fadvise_nocache],write 14761984~8192 [fadvise_sequential+fadvise_nocache],write 14774272~8192 [fadvise_sequential+fadvise_nocache],write 14786560~8192 [fadvise_sequential+fadvise_nocache],write 14798848~6453 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.705159028000001, 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058685117000000002, 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 44651 2024-04-24T00:35:18.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884304+0000", 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.884307+0000", 2024-04-24T00:35:18.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0850000000000001e-06 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886333+0000", 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020258310000000001 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886370+0000", 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6485000000000001e-05 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886385+0000", 2024-04-24T00:35:18.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5897e-05 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.886564+0000", 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178455 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.887161+0000", 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059714499999999999 2024-04-24T00:35:18.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.888890+0000", 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001729349 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920023+0000", 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031132238999999999 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:18.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942736+0000", 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02271306 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942884+0000", 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014812199999999999 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.942989+0000", 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105449 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46434 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578656+0000", 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010806742, 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026471701, 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46434 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578656+0000", 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578656+0000", 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578657+0000", 2024-04-24T00:35:18.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5000000000000001e-07 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578663+0000", 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5500000000000002e-06 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578684+0000", 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1169e-05 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578702+0000", 2024-04-24T00:35:18.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7390000000000001e-05 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.604458+0000", 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025756366999999999 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.604519+0000", 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0696999999999998e-05 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.605128+0000", 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060957800000000005 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46435 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578719+0000", 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010744066000001, 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027308378000000001, 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46435 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578719+0000", 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578719+0000", 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578720+0000", 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578725+0000", 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9559999999999996e-06 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578744+0000", 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9173000000000001e-05 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578759+0000", 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4596999999999999e-05 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.605215+0000", 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026456218 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.605342+0000", 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012719700000000001 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606028+0000", 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068533699999999995 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46436 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578777+0000", 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010686664000001, 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028104427000000001, 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46436 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578777+0000", 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578777+0000", 2024-04-24T00:35:18.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578778+0000", 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578783+0000", 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9699999999999998e-06 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578801+0000", 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8525999999999998e-05 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578819+0000", 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7569999999999999e-05 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606119+0000", 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027300843000000002 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606247+0000", 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012771700000000001 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606881+0000", 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063380800000000003 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46437 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578834+0000", 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010629449, 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029803969999999999, 2024-04-24T00:35:18.633 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46437 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578834+0000", 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578834+0000", 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578835+0000", 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3399999999999997e-07 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578840+0000", 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8760000000000004e-06 2024-04-24T00:35:18.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578858+0000", 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.861e-05 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578870+0000", 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2217e-05 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.606974+0000", 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028103633999999999 2024-04-24T00:35:18.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.607198+0000", 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000224019 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.608638+0000", 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00143968 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.637 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46438 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578888+0000", 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010575046, 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031231299000000001, 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46438 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578888+0000", 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578888+0000", 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578889+0000", 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.16e-07 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578894+0000", 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8729999999999998e-06 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578912+0000", 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8499999999999999e-05 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578925+0000", 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.225e-05 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.608693+0000", 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029768467 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.608780+0000", 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7012e-05 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.610119+0000", 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013392809999999999 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46439 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578942+0000", 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010521374, 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032570504, 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46439 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578942+0000", 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578942+0000", 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578943+0000", 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8999999999999995e-07 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578947+0000", 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7199999999999997e-06 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578966+0000", 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:35:18.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578978+0000", 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2113000000000001e-05 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.610203+0000", 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031225148000000001 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.610368+0000", 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016536400000000001 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.611512+0000", 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011438590000000001 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46440 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.646 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.578995+0000", 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010468084999999, 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033872845999999998, 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46440 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578995+0000", 2024-04-24T00:35:18.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578995+0000", 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.578996+0000", 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5300000000000003e-07 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579001+0000", 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6700000000000002e-06 2024-04-24T00:35:18.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579019+0000", 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8272999999999999e-05 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579031+0000", 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1744e-05 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.611603+0000", 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032572469999999999 2024-04-24T00:35:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.611738+0000", 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013527299999999999 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.612868+0000", 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011295630000000001 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46465 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 401408~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579084+0000", 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010379285999999, 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035121767999999998, 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46465 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579084+0000", 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579084+0000", 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3659999999999999e-06 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579092+0000", 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.973e-06 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579113+0000", 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1123000000000001e-05 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579138+0000", 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4807000000000002e-05 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.612958+0000", 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033820173000000002 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.613094+0000", 2024-04-24T00:35:18.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135544 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.614206+0000", 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001111782 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46466 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 417792~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.655 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579162+0000", 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010301437999999, 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036326009999999999, 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46466 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579162+0000", 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579162+0000", 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579163+0000", 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.066e-06 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579169+0000", 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7400000000000001e-06 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579186+0000", 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7694e-05 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579201+0000", 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4646e-05 2024-04-24T00:35:18.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.614296+0000", 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035094822999999997 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.614430+0000", 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134661 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.615488+0000", 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010573799999999999 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46467 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 434176~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579220+0000", 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010243409000001, 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036809401999999998, 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.660 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46467 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579220+0000", 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579220+0000", 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579221+0000", 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0529999999999999e-06 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579226+0000", 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4029999999999998e-06 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579268+0000", 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1498999999999997e-05 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5104e-05 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.615569+0000", 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036286232000000002 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.615643+0000", 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3569e-05 2024-04-24T00:35:18.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.616029+0000", 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038654200000000003 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46468 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 450560~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579352+0000", 2024-04-24T00:35:18.664 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010110818000001, 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037665068000000003, 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46468 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579352+0000", 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579352+0000", 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579353+0000", 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.33e-07 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0239999999999997e-06 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579375+0000", 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6432000000000001e-05 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579385+0000", 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.012e-05 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.616131+0000", 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036745932000000002 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.616288+0000", 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015730400000000001 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617017+0000", 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072932300000000004 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46469 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 466944~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579398+0000", 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010065661999999, 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038520933, 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46469 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.669 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579398+0000", 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579398+0000", 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579398+0000", 2024-04-24T00:35:18.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6600000000000005e-07 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579402+0000", 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3739999999999998e-06 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579414+0000", 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1956e-05 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579421+0000", 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2660000000000004e-06 2024-04-24T00:35:18.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617129+0000", 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037707683999999998 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617219+0000", 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0223000000000007e-05 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617918+0000", 2024-04-24T00:35:18.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069956399999999996 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46470 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 483328~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579430+0000", 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.010033273000001, 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039293489000000001, 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46470 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579430+0000", 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579430+0000", 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579431+0000", 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9999999999999997e-07 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579434+0000", 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.146e-06 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579448+0000", 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4344e-05 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579461+0000", 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2836e-05 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.617968+0000", 2024-04-24T00:35:18.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038507118999999999 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.618038+0000", 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9498000000000004e-05 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.618723+0000", 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068584600000000005 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.677 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46471 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 499712~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579477+0000", 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.009986400999999, 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040936221000000002, 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46471 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579477+0000", 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579477+0000", 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579478+0000", 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8999999999999995e-07 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579481+0000", 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5470000000000001e-06 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579491+0000", 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7829999999999993e-06 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579498+0000", 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3100000000000003e-06 2024-04-24T00:35:18.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.618828+0000", 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03932973 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.618984+0000", 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015575800000000001 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.620413+0000", 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001429303 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.681 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:46472 3.6 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:1d [read 516096~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:00.579513+0000", 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.009949940999999, 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041758379999999998, 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:18.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46472 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579513+0000", 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579513+0000", 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:18.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579514+0000", 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1999999999999998e-07 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579519+0000", 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9969999999999998e-06 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:18.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6002e-05 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4800000000000004e-06 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.620452+0000", 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040909214999999999 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.620562+0000", 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010995800000000001 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:00.621272+0000", 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070980800000000003 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:18.686 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:18.687 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:35:18.688 INFO:teuthology.orchestra.run.smithi064.stdout:2585 write 0xc4c7f80 thru 0xc4cbedf (0x3f60 bytes) 2024-04-24T00:35:18.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:18.690 INFO:teuthology.orchestra.run.smithi064.stdout:2586 read 0x6acd17f thru 0x6ad3963 (0x67e5 bytes) 2024-04-24T00:35:18.690 INFO:teuthology.orchestra.run.smithi064.stdout:2587 punch from 0x3f7b623 to 0x3f85ad5, (0xa4b2 bytes) 2024-04-24T00:35:18.691 INFO:teuthology.orchestra.run.smithi064.stdout:2588 write 0xb1f9d96 thru 0xb204c3a (0xaea5 bytes) 2024-04-24T00:35:18.691 INFO:teuthology.orchestra.run.smithi064.stdout:2589 read 0x1f23b5 thru 0x1fe292 (0xbede bytes) 2024-04-24T00:35:18.691 INFO:teuthology.orchestra.run.smithi064.stdout:2590 trunc from 0xc4cbee0 to 0x219e8f5 2024-04-24T00:35:18.691 INFO:teuthology.orchestra.run.smithi064.stdout:2591 read 0x1027872 thru 0x102d6c8 (0x5e57 bytes) 2024-04-24T00:35:18.691 INFO:teuthology.orchestra.run.smithi064.stdout:2592 trunc from 0x219e8f5 to 0x551340b 2024-04-24T00:35:18.691 INFO:teuthology.orchestra.run.smithi064.stdout:2593 trunc from 0x551340b to 0x37fe92c 2024-04-24T00:35:18.691 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:18.681+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:18.697 INFO:teuthology.orchestra.run.smithi064.stdout:2594 read 0x327893f thru 0x327ed29 (0x63eb bytes) 2024-04-24T00:35:18.740 INFO:teuthology.orchestra.run.smithi064.stdout:2595 writesame 0xe502a43 thru 0xe5126f7 data_size 0x1(0xfcb5 bytes) 2024-04-24T00:35:18.740 INFO:teuthology.orchestra.run.smithi064.stdout:2596 read 0x182774b thru 0x1833d74 (0xc62a bytes) 2024-04-24T00:35:18.740 INFO:teuthology.orchestra.run.smithi064.stdout:2598 punch from 0x88d2d6d to 0x88daa29, (0x7cbc bytes) 2024-04-24T00:35:18.758 INFO:teuthology.orchestra.run.smithi064.stdout:2599 trunc from 0xe5126f8 to 0x5855996 2024-04-24T00:35:18.777 INFO:teuthology.orchestra.run.smithi064.stdout:2601 trunc from 0x5855996 to 0x706ea27 2024-04-24T00:35:18.789 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:18.783+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:18.791 INFO:teuthology.orchestra.run.smithi064.stdout:2602 punch from 0x4c983ab to 0x4c9c535, (0x418a bytes) 2024-04-24T00:35:18.791 INFO:teuthology.orchestra.run.smithi064.stdout:2604 read 0x4a8096c thru 0x4a83846 (0x2edb bytes) 2024-04-24T00:35:18.791 INFO:teuthology.orchestra.run.smithi064.stdout:2605 read 0x2047d03 thru 0x204a04e (0x234c bytes) 2024-04-24T00:35:18.792 INFO:teuthology.orchestra.run.smithi064.stdout:2606 punch from 0x29f55ca to 0x2a00280, (0xacb6 bytes) 2024-04-24T00:35:18.792 INFO:teuthology.orchestra.run.smithi064.stdout:2607 read 0x1ae59d2 thru 0x1aec819 (0x6e48 bytes) 2024-04-24T00:35:18.792 INFO:teuthology.orchestra.run.smithi064.stdout:2608 punch from 0x64466bc to 0x644cd95, (0x66d9 bytes) 2024-04-24T00:35:18.792 INFO:teuthology.orchestra.run.smithi064.stdout:2609 writesame 0x520a0ea thru 0x5215712 data_size 0x1(0xb629 bytes) 2024-04-24T00:35:18.835 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:18.835 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:18.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:18.889 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:18.883+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:18.985 INFO:teuthology.orchestra.run.smithi064.stdout:2610 write 0x9358ed0 thru 0x9368a19 (0xfb4a bytes) 2024-04-24T00:35:18.990 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:18.985+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:19.014 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:19.014 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:19.015 INFO:teuthology.orchestra.run.smithi064.stdout:2613 writesame 0xe9719e2 thru 0xe97db56 data_size 0x23(0xc175 bytes) 2024-04-24T00:35:19.015 INFO:teuthology.orchestra.run.smithi064.stdout:2614 write 0xabf6fbd thru 0xac06709 (0xf74d bytes) 2024-04-24T00:35:19.029 INFO:teuthology.orchestra.run.smithi064.stdout:2615 write 0x6d9811b thru 0x6da2aee (0xa9d4 bytes) 2024-04-24T00:35:19.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:19.040 INFO:teuthology.orchestra.run.smithi064.stdout:2616 write 0x2604d89 thru 0x2614418 (0xf690 bytes) 2024-04-24T00:35:19.053 INFO:teuthology.orchestra.run.smithi064.stdout:2617 clone 19 order 23 su 65536 sc 15 2024-04-24T00:35:19.091 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:19.085+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:19.166 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.518599480000006, 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20552 3.9 e49/21 3:923662ec:::rbd_data.115ee4742374.000000000000000a:head v 49'88, mlcod=48'86)", 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.189905879999998, 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062734230000000002, 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973502+0000", 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.04e-06 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973677+0000", 2024-04-24T00:35:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473200000000001 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973714+0000", 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7117999999999999e-05 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973734+0000", 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0102000000000002e-05 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973879+0000", 2024-04-24T00:35:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145654 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973946+0000", 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6476999999999994e-05 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979687+0000", 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057411140000000003 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979773+0000", 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6186000000000005e-05 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.580324623999999, 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:35:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.580250769000003, 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:35:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:35:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:35:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.883555985000001, 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:35:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.078024026000001, 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:35:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:68687 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'253, mlcod=94'249)", 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.4980125830000004, 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005827128, 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665394+0000", 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7700000000000004e-07 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665402+0000", 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4030000000000007e-06 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665414+0000", 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1467999999999999e-05 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665419+0000", 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1780000000000002e-06 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665485+0000", 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5944000000000004e-05 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665507+0000", 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2359999999999999e-05 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.671176+0000", 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056689130000000003 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.671220+0000", 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4184999999999998e-05 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:69871 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'404, mlcod=94'396)", 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.3309352560000001, 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055042449999999996, 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832471+0000", 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9200000000000001e-07 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832475+0000", 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.873e-06 2024-04-24T00:35:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832482+0000", 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832486+0000", 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2470000000000002e-06 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832537+0000", 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1233000000000002e-05 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832563+0000", 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5269000000000002e-05 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.837951+0000", 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053882720000000004 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.837975+0000", 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3683e-05 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:69977 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'424, mlcod=94'419)", 2024-04-24T00:35:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.3111792119999999, 2024-04-24T00:35:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056478099999999996, 2024-04-24T00:35:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852227+0000", 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852231+0000", 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1489999999999996e-06 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852240+0000", 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0010000000000007e-06 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852245+0000", 2024-04-24T00:35:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.498e-06 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852283+0000", 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8195000000000003e-05 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852298+0000", 2024-04-24T00:35:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4911e-05 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.857827+0000", 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055296210000000002 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.857874+0000", 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6897999999999997e-05 2024-04-24T00:35:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70740 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'523, mlcod=94'518)", 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1980256100000002, 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061709410000000001, 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965381+0000", 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.094e-06 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965389+0000", 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9580000000000007e-06 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965423+0000", 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3787e-05 2024-04-24T00:35:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965443+0000", 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0390999999999998e-05 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965583+0000", 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013995500000000001 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965606+0000", 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3003999999999999e-05 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971482+0000", 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058757649999999998 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971551+0000", 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8986999999999995e-05 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70743 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'524, mlcod=94'518)", 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1973212819999999, 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055099820000000001, 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966086+0000", 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1969999999999999e-06 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966096+0000", 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.995e-06 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966113+0000", 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7538e-05 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966124+0000", 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966181+0000", 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6937000000000001e-05 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966199+0000", 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8002000000000001e-05 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971556+0000", 2024-04-24T00:35:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053569029999999997 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971594+0000", 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8186999999999999e-05 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70750 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'527, mlcod=94'519)", 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1950447110000004, 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005844307, 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968362+0000", 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4900000000000004e-07 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968371+0000", 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.619e-06 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968388+0000", 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7113999999999999e-05 2024-04-24T00:35:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968401+0000", 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3495e-05 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968434+0000", 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3306000000000003e-05 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968446+0000", 2024-04-24T00:35:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1668000000000001e-05 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.974121+0000", 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056745290000000002 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.974205+0000", 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4627000000000001e-05 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70759 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'530, mlcod=94'521)", 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1923338680000004, 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054257760000000002, 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971073+0000", 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4300000000000002e-07 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971078+0000", 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971086+0000", 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4400000000000005e-06 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971091+0000", 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.763e-06 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971138+0000", 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6897000000000002e-05 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971150+0000", 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2480000000000001e-05 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.976394+0000", 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052434580000000003 2024-04-24T00:35:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.976498+0000", 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103973 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70804 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'539, mlcod=94'532)", 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.182746485, 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058417199999999999, 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980660+0000", 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.99e-07 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980665+0000", 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.95e-06 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980681+0000", 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6653000000000001e-05 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980686+0000", 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9420000000000003e-06 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980873+0000", 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018610100000000001 2024-04-24T00:35:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980889+0000", 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6884e-05 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986449+0000", 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055591700000000004 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986501+0000", 2024-04-24T00:35:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2321000000000003e-05 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70823 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'542, mlcod=94'536)", 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1801163910000003, 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057985090000000003, 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983290+0000", 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983296+0000", 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7969999999999997e-06 2024-04-24T00:35:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983304+0000", 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4519999999999997e-06 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983309+0000", 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6809999999999997e-06 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983353+0000", 2024-04-24T00:35:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3844999999999998e-05 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983365+0000", 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.235e-05 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.988997+0000", 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005632176 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.989088+0000", 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0587999999999994e-05 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70843 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'546, mlcod=94'537)", 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1773795590000002, 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059306039999999999, 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986027+0000", 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2e-07 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986031+0000", 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986039+0000", 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1880000000000002e-06 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986043+0000", 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2339999999999996e-06 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986133+0000", 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9783000000000004e-05 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986146+0000", 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2816000000000001e-05 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.991875+0000", 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057291429999999999 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.991957+0000", 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2027999999999995e-05 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70942 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'565, mlcod=94'560)", 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1588348310000001, 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005499884, 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004572+0000", 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.576e-06 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004583+0000", 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0383e-05 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004603+0000", 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9738999999999999e-05 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004612+0000", 2024-04-24T00:35:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6919999999999997e-06 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004691+0000", 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8771999999999997e-05 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004711+0000", 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0018e-05 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009995+0000", 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052835520000000004 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.010071+0000", 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6151999999999999e-05 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70944 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'567, mlcod=94'561)", 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1564802910000003, 2024-04-24T00:35:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059596270000000003, 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006927+0000", 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006936+0000", 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6719999999999999e-06 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006960+0000", 2024-04-24T00:35:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4249000000000001e-05 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006975+0000", 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4259e-05 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.007069+0000", 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4477999999999999e-05 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.007088+0000", 2024-04-24T00:35:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8332999999999999e-05 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012838+0000", 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057502830000000001 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012885+0000", 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7206999999999997e-05 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70966 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'570, mlcod=94'563)", 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1541580380000003, 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059298320000000003, 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009249+0000", 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.155e-06 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009257+0000", 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.892e-06 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009275+0000", 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8272000000000001e-05 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009288+0000", 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3107e-05 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009373+0000", 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4561999999999994e-05 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009407+0000", 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4100000000000002e-05 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.015084+0000", 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056774750000000004 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.015177+0000", 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3269000000000004e-05 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70979 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'574, mlcod=94'565)", 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1511587329999999, 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054674650000000003, 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012248+0000", 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0920000000000001e-06 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012256+0000", 2024-04-24T00:35:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9119999999999998e-06 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012273+0000", 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7280000000000001e-05 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012295+0000", 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1866999999999999e-05 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012370+0000", 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5278999999999995e-05 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012409+0000", 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8226999999999998e-05 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.017649+0000", 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052406350000000004 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.017714+0000", 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5172999999999994e-05 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.258 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:19.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:19.260 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:19.253+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:19.360 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:19.355+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:19.432 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:19.432 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:19.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:19.494 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:19.489+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:19.553 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:35:19.560 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:19.553+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:19.579 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:19.580 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:19.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:19.685 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:19.681+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:19.762 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:19.757+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:19.767 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.234428531999995, 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:35:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.234404617999999, 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:35:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:35:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:35:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.234377444000003, 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:35:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.234351339, 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:35:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.233940357999998, 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:35:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:35:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:35:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.233920549000004, 2024-04-24T00:35:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:35:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.233867489999994, 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:35:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.233832633999995, 2024-04-24T00:35:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.233811070000002, 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:35:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.233781578000006, 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:35:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:35:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.233760716000006, 2024-04-24T00:35:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:35:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:35:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.233738142000007, 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:35:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:35:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.551478643999999, 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:35:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:35:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:35:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.551127729000001, 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:35:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.544239408999999, 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:35:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44653 3.c e79/56 3:3545ecd3:::rbd_data.115ec7ed39de.0000000000000002:head v 79'268, mlcod=77'266)", 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.873495687999998, 2024-04-24T00:35:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022668765, 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7800000000000002e-07 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892860+0000", 2024-04-24T00:35:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003115821 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892903+0000", 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3176999999999997e-05 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892926+0000", 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2459999999999998e-05 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.893026+0000", 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010062 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.893059+0000", 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2806999999999999e-05 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.912296+0000", 2024-04-24T00:35:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019237431999999999 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.912412+0000", 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011597 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44656 3.7 e79/44 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:head v 79'103, mlcod=77'101)", 2024-04-24T00:35:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.868912943000002, 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026507441999999999, 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894327+0000", 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1e-07 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898194+0000", 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038672889999999999 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898203+0000", 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6340000000000007e-06 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898208+0000", 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2009999999999998e-06 2024-04-24T00:35:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898262+0000", 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4466999999999999e-05 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898280+0000", 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7492000000000001e-05 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920764+0000", 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022483974 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920834+0000", 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0174999999999998e-05 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44661 3.17 e79/59 3:eff1c653:::rbd_data.115ec7ed39de.0000000000000005:head v 79'91, mlcod=77'89)", 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.867045654000002, 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024065629000000002, 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8299999999999996e-07 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897771+0000", 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015761690000000001 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897786+0000", 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5051999999999999e-05 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897794+0000", 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4689999999999997e-06 2024-04-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897869+0000", 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.47e-05 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897899+0000", 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9777e-05 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920181+0000", 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022282560999999999 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920259+0000", 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8218000000000002e-05 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.285957783000001, 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:35:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:35:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.275681367000001, 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:19.880 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:19.881 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:19.882 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:19.877+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:19.982 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:19.979+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:20.014 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:20.015 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:20.015 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:20.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:20.083 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:20.077+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:20.184 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:20.179+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:20.192 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:20.192 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:20.192 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:20.193 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:20.193 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:20.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:20.294 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:20.291+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.730532036, 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:35:20.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.380 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.730455836999994, 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:20.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:35:20.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.384 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.729997663000006, 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.388 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.729940356, 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.389 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:35:20.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:35:20.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.729681389999996, 2024-04-24T00:35:20.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:35:20.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:20.397 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.728759917999994, 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.401 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.156945187999995, 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.402 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.407 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.141415938999998, 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:20.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:20.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:20.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:20.413 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.413 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.413 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.413 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:20.413 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.512692619000006, 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:20.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.497657492000002, 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:35:20.419 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:35:20.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:35:20.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.735159494000001, 2024-04-24T00:35:20.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:20.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.721336536000003, 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:20.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.494195351000002, 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:20.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:20.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:20.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:20.437 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.482405700999998, 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30704 3.1a 3:5e74e97a:::rbd_data.115e8210e51a.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:20.441 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.118913782, 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043351496000000003, 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30704 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:20.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252358+0000", 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1790000000000001e-06 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252364+0000", 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6549999999999998e-06 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252381+0000", 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6767000000000001e-05 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252390+0000", 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9959999999999999e-06 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252492+0000", 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102563 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252668+0000", 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017553499999999999 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.254545+0000", 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001877037 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295422+0000", 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040876527000000003 2024-04-24T00:35:20.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295513+0000", 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1838999999999999e-05 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295614+0000", 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010027900000000001 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295708+0000", 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4118999999999998e-05 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.488365637000001, 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.447 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:20.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:20.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33518 3.b e70/59 ondisk, result = 0)", 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.477042109000003, 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042187448000000002, 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894230+0000", 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7680000000000001e-06 2024-04-24T00:35:20.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894239+0000", 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3319999999999998e-06 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894266+0000", 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049000000000001e-05 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894281+0000", 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5245000000000001e-05 2024-04-24T00:35:20.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936247+0000", 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041965833000000001 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936257+0000", 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936415+0000", 2024-04-24T00:35:20.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158605 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33734 3.b 3:d4ebe596:::rbd_data.115ec4f9b3a7.000000000000000e:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.851999913999997, 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041668288999999997, 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.457 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33734 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519272+0000", 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9870000000000002e-06 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519282+0000", 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0203e-05 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519310+0000", 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7880000000000001e-05 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519324+0000", 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519402+0000", 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7865000000000001e-05 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519481+0000", 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9103000000000005e-05 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.522670+0000", 2024-04-24T00:35:20.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003189 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560812+0000", 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038141427999999998 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560831+0000", 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9820999999999999e-05 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560886+0000", 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5087999999999998e-05 2024-04-24T00:35:20.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560938+0000", 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1993999999999997e-05 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44651 3.6 e79/59 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head v 79'72, mlcod=77'70)", 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.482292546, 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048315631999999997, 2024-04-24T00:35:20.462 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892459+0000", 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034802259999999999 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892475+0000", 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5998000000000002e-05 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892485+0000", 2024-04-24T00:35:20.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8679999999999994e-06 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892653+0000", 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016800800000000001 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892721+0000", 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8953000000000002e-05 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937243+0000", 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044521666000000001 2024-04-24T00:35:20.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937293+0000", 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0043000000000002e-05 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44659 3.1d e79/53 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head v 79'105, mlcod=77'103)", 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.474811363000001, 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038358476000000002, 2024-04-24T00:35:20.466 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896460+0000", 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2399999999999997e-07 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897721+0000", 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012617049999999999 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897744+0000", 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2982000000000001e-05 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897756+0000", 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1559999999999999e-05 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897836+0000", 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0434999999999998e-05 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897871+0000", 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5147000000000001e-05 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:20.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934696+0000", 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036824511999999997 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934817+0000", 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121311 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.470 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:20.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:20.472 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:20.469+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:20.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:20.572 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:20.567+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:20.632 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:20.644 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:20.644 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72658 3.16 3.385fc8d6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.641037+0000", 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0022444969999999998, 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022853700000000001, 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "op must wait for map", 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72658 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.641037+0000", 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.641037+0000", 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.641040+0000", 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.559e-06 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:20.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.641050+0000", 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8449999999999998e-06 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.641073+0000", 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3357e-05 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.641085+0000", 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1554e-05 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:20.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.641195+0000", 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110031 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.641205+0000", 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0823e-05 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:35:20.648 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:20.652 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:20.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:20.673 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:20.665+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:20.676 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:20.677 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:20.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:20.717 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:20.717 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:20.744 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:20.744 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:20.767 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:20.768 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:20.774 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:20.769+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:20.794 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:20.794 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:20.875 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:20.871+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:20.877 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:20.877 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:20.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:20.978 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:20.975+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:21.052 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:21.052 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:21.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:21.077 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:21.069+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:21.178 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:21.175+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:21.222 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:21.222 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:21.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:21.278 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:21.273+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:21.380 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:21.375+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:21.395 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:21.395 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:21.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:21.498 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:21.495+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:21.576 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:21.576 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:21.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:35:21.582 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:21.579+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:21.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:21.683 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:21.679+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:21.757 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:21.757 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:21.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:21.814 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:21.811+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:21.833 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-04-24T00:35:21.885 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:21.881+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:21.952 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:21.952 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:21.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:22.010 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:22.007+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:22.070 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:22.070 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:22.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:22.086 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:22.081+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:22.184 INFO:teuthology.orchestra.run.smithi064.stdout:2618 writesame 0xcab98b5 thru 0xcac6cba data_size 0x7(0xd406 bytes) 2024-04-24T00:35:22.187 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:22.181+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:22.199 INFO:teuthology.orchestra.run.smithi064.stdout:2619 writesame 0x8ccaaec thru 0x8ccdff8 data_size 0x9(0x350d bytes) 2024-04-24T00:35:22.204 INFO:teuthology.orchestra.run.smithi064.stdout:2620 read 0x8f761f7 thru 0x8f85999 (0xf7a3 bytes) 2024-04-24T00:35:22.205 INFO:teuthology.orchestra.run.smithi064.stdout:2622 write 0x9167554 thru 0x916e84e (0x72fb bytes) 2024-04-24T00:35:22.214 INFO:teuthology.orchestra.run.smithi064.stdout:2623 writesame 0xcb2da78 thru 0xcb2f219 data_size 0x1(0x17a2 bytes) 2024-04-24T00:35:22.215 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:22.215 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:22.222 INFO:teuthology.orchestra.run.smithi064.stdout:2624 write 0x11075b9 thru 0x110d60d (0x6055 bytes) 2024-04-24T00:35:22.228 INFO:teuthology.orchestra.run.smithi064.stdout:2627 writesame 0x5738979 thru 0x573aec9 data_size 0x1(0x2551 bytes) 2024-04-24T00:35:22.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:22.241 INFO:teuthology.orchestra.run.smithi064.stdout:2628 read 0x2ce283b thru 0x2ce2dc0 (0x586 bytes) 2024-04-24T00:35:22.243 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:22.243 INFO:teuthology.orchestra.run.smithi064.stdout:2630 writesame 0x4fd369c thru 0x4fd434f data_size 0x6(0xcb4 bytes) 2024-04-24T00:35:22.253 INFO:teuthology.orchestra.run.smithi064.stdout:2631 trunc from 0xe97db57 to 0xa356f4d 2024-04-24T00:35:22.273 INFO:teuthology.orchestra.run.smithi064.stdout:2632 read 0x11102c2 thru 0x1118034 (0x7d73 bytes) 2024-04-24T00:35:22.279 INFO:teuthology.orchestra.run.smithi064.stdout:2633 trunc from 0xa356f4d to 0x687b150 2024-04-24T00:35:22.287 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:22.281+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:22.309 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:22.309 INFO:teuthology.orchestra.run.smithi064.stdout:2636 read 0x3b51685 thru 0x3b5a5a8 (0x8f24 bytes) 2024-04-24T00:35:22.309 INFO:teuthology.orchestra.run.smithi064.stdout:2637 read 0x1389a5c thru 0x1397ba3 (0xe148 bytes) 2024-04-24T00:35:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:2638 trunc from 0x687b150 to 0x8bab965 2024-04-24T00:35:22.335 INFO:teuthology.orchestra.run.smithi064.stdout:2639 trunc from 0x8bab965 to 0x2db0c9d 2024-04-24T00:35:22.359 INFO:teuthology.orchestra.run.smithi064.stdout:2640 punch from 0x171d75d to 0x172480f, (0x70b2 bytes) 2024-04-24T00:35:22.359 INFO:teuthology.orchestra.run.smithi064.stdout:2642 read 0x29c86ab thru 0x29cd4a7 (0x4dfd bytes) 2024-04-24T00:35:22.359 INFO:teuthology.orchestra.run.smithi064.stdout:2643 read 0x24a75e7 thru 0x24ae1c4 (0x6bde bytes) 2024-04-24T00:35:22.361 INFO:teuthology.orchestra.run.smithi064.stdout:2646 write 0x200ed4f thru 0x201e8ba (0xfb6c bytes) 2024-04-24T00:35:22.371 INFO:teuthology.orchestra.run.smithi064.stdout:2647 punch from 0x1ef36b8 to 0x1effa52, (0xc39a bytes) 2024-04-24T00:35:22.376 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:22.376 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:22.388 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:22.381+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:22.394 INFO:teuthology.orchestra.run.smithi064.stdout:2649 writesame 0x96cd170 thru 0x96cd6e8 data_size 0x1(0x579 bytes) 2024-04-24T00:35:22.394 INFO:teuthology.orchestra.run.smithi064.stdout:2650 write 0x72298d6 thru 0x722dea6 (0x45d1 bytes) 2024-04-24T00:35:22.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:22.396 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:22.396 INFO:teuthology.orchestra.run.smithi064.stdout:2653 write 0x3b21d26 thru 0x3b264c6 (0x47a1 bytes) 2024-04-24T00:35:22.401 INFO:teuthology.orchestra.run.smithi064.stdout:2654 read 0x6c9838e thru 0x6c9ecca (0x693d bytes) 2024-04-24T00:35:22.402 INFO:teuthology.orchestra.run.smithi064.stdout:2657 punch from 0x134fc81 to 0x13542f6, (0x4675 bytes) 2024-04-24T00:35:22.415 INFO:teuthology.orchestra.run.smithi064.stdout:2658 write 0xad584a9 thru 0xad67c04 (0xf75c bytes) 2024-04-24T00:35:22.425 INFO:teuthology.orchestra.run.smithi064.stdout:2660 trunc from 0xad67c05 to 0xa241b5b 2024-04-24T00:35:22.439 INFO:teuthology.orchestra.run.smithi064.stdout:2662 read 0x5dc6cd3 thru 0x5dd3e54 (0xd182 bytes) 2024-04-24T00:35:22.440 INFO:teuthology.orchestra.run.smithi064.stdout:2663 read 0x25239e6 thru 0x252aefd (0x7518 bytes) 2024-04-24T00:35:22.441 INFO:teuthology.orchestra.run.smithi064.stdout:2664 read 0x407439e thru 0x4078b37 (0x479a bytes) 2024-04-24T00:35:22.442 INFO:teuthology.orchestra.run.smithi064.stdout:2665 punch from 0x1d44962 to 0x1d4c0f5, (0x7793 bytes) 2024-04-24T00:35:22.442 INFO:teuthology.orchestra.run.smithi064.stdout:2666 writesame 0x2043165 thru 0x204b485 data_size 0x1(0x8321 bytes) 2024-04-24T00:35:22.489 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:22.481+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:22.521 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:22.521 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:22.522 INFO:teuthology.orchestra.run.smithi064.stdout:2667 write 0x846fb19 thru 0x84742e6 (0x47ce bytes) 2024-04-24T00:35:22.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:22.550 INFO:teuthology.orchestra.run.smithi064.stdout:2668 write 0xd721ced thru 0xd7285ad (0x68c1 bytes) 2024-04-24T00:35:22.556 INFO:teuthology.orchestra.run.smithi064.stdout:2669 writesame 0x19f095b thru 0x19f6fa6 data_size 0x2(0x664c bytes) 2024-04-24T00:35:22.567 INFO:teuthology.orchestra.run.smithi064.stdout:2670 read 0xa2efc24 thru 0xa2f414e (0x452b bytes) 2024-04-24T00:35:22.568 INFO:teuthology.orchestra.run.smithi064.stdout:2671 write 0x3aa9b4c thru 0x3ab0686 (0x6b3b bytes) 2024-04-24T00:35:22.573 INFO:teuthology.orchestra.run.smithi064.stdout:2672 read 0xd5f691c thru 0xd6027ee (0xbed3 bytes) 2024-04-24T00:35:22.574 INFO:teuthology.orchestra.run.smithi064.stdout:2673 writesame 0xc2c6d0a thru 0xc2d2d00 data_size 0x1(0xbff7 bytes) 2024-04-24T00:35:22.590 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:22.585+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:22.646 INFO:teuthology.orchestra.run.smithi064.stdout:2674 trunc from 0xd7285ae to 0x32a5f68 2024-04-24T00:35:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:35:22.684 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:22.684 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:22.690 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:22.685+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:22.692 INFO:teuthology.orchestra.run.smithi064.stdout:2675 writesame 0x21e48ec thru 0x21f1053 data_size 0x24(0xc768 bytes) 2024-04-24T00:35:22.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:22.698 INFO:teuthology.orchestra.run.smithi064.stdout:2676 read 0x3d7bb2 thru 0x3d892e (0xd7d bytes) 2024-04-24T00:35:22.699 INFO:teuthology.orchestra.run.smithi064.stdout:2678 read 0x33258 thru 0x3ebf4 (0xb99d bytes) 2024-04-24T00:35:22.700 INFO:teuthology.orchestra.run.smithi064.stdout:2679 write 0xfd5d6c thru 0xfda3ae (0x4643 bytes) 2024-04-24T00:35:22.720 INFO:teuthology.orchestra.run.smithi064.stdout:2680 write 0x7623482 thru 0x763053c (0xd0bb bytes) 2024-04-24T00:35:22.723 INFO:teuthology.orchestra.run.smithi064.stdout:2681 read 0x1eade7c thru 0x1eb7459 (0x95de bytes) 2024-04-24T00:35:22.729 INFO:teuthology.orchestra.run.smithi064.stdout:2682 trunc from 0x763053d to 0xc1f03a5 2024-04-24T00:35:22.741 INFO:teuthology.orchestra.run.smithi064.stdout:2684 trunc from 0xc1f03a5 to 0xd247fe5 2024-04-24T00:35:22.751 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:22.751 INFO:teuthology.orchestra.run.smithi064.stdout:2686 writesame 0x41c96fc thru 0x41d5b65 data_size 0x1f(0xc46a bytes) 2024-04-24T00:35:22.757 INFO:teuthology.orchestra.run.smithi064.stdout:2688 read 0xba6a4ac thru 0xba6e26c (0x3dc1 bytes) 2024-04-24T00:35:22.757 INFO:teuthology.orchestra.run.smithi064.stdout:2690 read 0xce8c546 thru 0xce91104 (0x4bbf bytes) 2024-04-24T00:35:22.758 INFO:teuthology.orchestra.run.smithi064.stdout:2691 trunc from 0xd247fe5 to 0xc40b0b2 2024-04-24T00:35:22.769 INFO:teuthology.orchestra.run.smithi064.stdout:2692 read 0xbc6f37f thru 0xbc7362f (0x42b1 bytes) 2024-04-24T00:35:22.769 INFO:teuthology.orchestra.run.smithi064.stdout:2693 write 0x964cadb thru 0x965a8be (0xdde4 bytes) 2024-04-24T00:35:22.780 INFO:teuthology.orchestra.run.smithi064.stdout:2694 write 0x9f4e349 thru 0x9f4f6ec (0x13a4 bytes) 2024-04-24T00:35:22.786 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:22.786 INFO:teuthology.orchestra.run.smithi064.stdout:2696 punch from 0xb4cd9ad to 0xb4da7ca, (0xce1d bytes) 2024-04-24T00:35:22.786 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:22.787 INFO:teuthology.orchestra.run.smithi064.stdout:2700 trunc from 0xc40b0b2 to 0xb579604 2024-04-24T00:35:22.791 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:22.785+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:22.801 INFO:teuthology.orchestra.run.smithi064.stdout:2701 write 0x8df504a thru 0x8dfe0b2 (0x9069 bytes) 2024-04-24T00:35:22.811 INFO:teuthology.orchestra.run.smithi064.stdout:2702 trunc from 0xb579604 to 0x74b4091 2024-04-24T00:35:22.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:22.822 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:22.822 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:22.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:22.851 INFO:teuthology.orchestra.run.smithi064.stdout:2703 write 0x9a762b7 thru 0x9a79169 (0x2eb3 bytes) 2024-04-24T00:35:22.855 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:22.866 INFO:teuthology.orchestra.run.smithi064.stdout:2706 trunc from 0x9a7916a to 0x31f4669 2024-04-24T00:35:22.889 INFO:teuthology.orchestra.run.smithi064.stdout:2707 read 0x64b92b thru 0x657bda (0xc2b0 bytes) 2024-04-24T00:35:22.889 INFO:teuthology.orchestra.run.smithi064.stdout:2708 read 0x1fc2689 thru 0x1fc4192 (0x1b0a bytes) 2024-04-24T00:35:22.892 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:22.887+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:22.914 INFO:teuthology.orchestra.run.smithi064.stdout:2710 writesame 0xdbee623 thru 0xdbee91c data_size 0x3(0x2fa bytes) 2024-04-24T00:35:22.915 INFO:teuthology.orchestra.run.smithi064.stdout:2711 read 0x7b030d9 thru 0x7b1029e (0xd1c6 bytes) 2024-04-24T00:35:22.915 INFO:teuthology.orchestra.run.smithi064.stdout:2712 write 0x6cfee1c thru 0x6cff3f8 (0x5dd bytes) 2024-04-24T00:35:22.917 INFO:teuthology.orchestra.run.smithi064.stdout:2713 writesame 0xa7794b thru 0xa83fa4 data_size 0x5d(0xc65a bytes) 2024-04-24T00:35:22.922 INFO:teuthology.orchestra.run.smithi064.stdout:2714 read 0x4103b28 thru 0x41100e6 (0xc5bf bytes) 2024-04-24T00:35:22.923 INFO:teuthology.orchestra.run.smithi064.stdout:2715 read 0x81311fd thru 0x8132e87 (0x1c8b bytes) 2024-04-24T00:35:22.923 INFO:teuthology.orchestra.run.smithi064.stdout:2716 write 0xcf76da5 thru 0xcf81e4e (0xb0aa bytes) 2024-04-24T00:35:22.931 INFO:teuthology.orchestra.run.smithi064.stdout:2717 writesame 0xd5c59b0 thru 0xd5d5174 data_size 0x1(0xf7c5 bytes) 2024-04-24T00:35:22.983 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:22.983 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:22.993 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:22.987+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:22.993 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:22.993 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:23.000 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:23.000 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:23.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:23.017 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:23.017 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:23.033 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:23.034 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:23.053 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:23.053 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:23.073 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:23.074 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:23.087 INFO:teuthology.orchestra.run.smithi064.stdout:2718 read 0x1561e63 thru 0x156a076 (0x8214 bytes) 2024-04-24T00:35:23.087 INFO:teuthology.orchestra.run.smithi064.stdout:2720 read 0xbc291c7 thru 0xbc2eafd (0x5937 bytes) 2024-04-24T00:35:23.094 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:23.087+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:23.095 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:23.095 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:23.096 INFO:teuthology.orchestra.run.smithi064.stdout:2721 trunc from 0xdbee91d to 0x7a5fcba 2024-04-24T00:35:23.106 INFO:teuthology.orchestra.run.smithi064.stdout:2723 read 0x7191fe4 thru 0x71a1a3f (0xfa5c bytes) 2024-04-24T00:35:23.106 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.106 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.106 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.107 INFO:teuthology.orchestra.run.smithi064.stdout:2731 write 0x1a7e3e7 thru 0x1a84fe2 (0x6bfc bytes) 2024-04-24T00:35:23.109 INFO:teuthology.orchestra.run.smithi064.stdout:2732 write 0x963d63 thru 0x96a4e0 (0x677e bytes) 2024-04-24T00:35:23.114 INFO:teuthology.orchestra.run.smithi064.stdout:2734 writesame 0x64a7ec0 thru 0x64addae data_size 0x1(0x5eef bytes) 2024-04-24T00:35:23.124 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:23.124 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:23.137 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:23.137 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:23.137 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:23.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:23.162 INFO:teuthology.orchestra.run.smithi064.stdout:2735 read 0x1c6575e thru 0x1c68fbd (0x3860 bytes) 2024-04-24T00:35:23.163 INFO:teuthology.orchestra.run.smithi064.stdout:2736 read 0x5083eeb thru 0x5088a00 (0x4b16 bytes) 2024-04-24T00:35:23.170 INFO:teuthology.orchestra.run.smithi064.stdout:2737 trunc from 0x7a5fcba to 0x39b8475 2024-04-24T00:35:23.194 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:23.189+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:23.194 INFO:teuthology.orchestra.run.smithi064.stdout:2739 trunc from 0x39b8475 to 0x3208316 2024-04-24T00:35:23.233 INFO:teuthology.orchestra.run.smithi064.stdout:2740 write 0xa7bb271 thru 0xa7be563 (0x32f3 bytes) 2024-04-24T00:35:23.238 INFO:teuthology.orchestra.run.smithi064.stdout:2743 punch from 0xe097a0 to 0xe109e1, (0x7241 bytes) 2024-04-24T00:35:23.238 INFO:teuthology.orchestra.run.smithi064.stdout:2744 read 0x7e48ca6 thru 0x7e5547d (0xc7d8 bytes) 2024-04-24T00:35:23.238 INFO:teuthology.orchestra.run.smithi064.stdout:2745 read 0x2b7ebe1 thru 0x2b888d6 (0x9cf6 bytes) 2024-04-24T00:35:23.239 INFO:teuthology.orchestra.run.smithi064.stdout:2746 read 0x9f5ac43 thru 0x9f6a1f8 (0xf5b6 bytes) 2024-04-24T00:35:23.239 INFO:teuthology.orchestra.run.smithi064.stdout:2748 write 0x738903d thru 0x738dad9 (0x4a9d bytes) 2024-04-24T00:35:23.243 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.252 INFO:teuthology.orchestra.run.smithi064.stdout:2751 trunc from 0xa7be564 to 0x6af7d09 2024-04-24T00:35:23.290 INFO:teuthology.orchestra.run.smithi064.stdout:2752 trunc from 0x6af7d09 to 0x93923fd 2024-04-24T00:35:23.294 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:23.289+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:23.296 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:23.296 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:23.296 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:23.297 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:23.297 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:23.304 INFO:teuthology.orchestra.run.smithi064.stdout:2753 read 0x4009d2b thru 0x400c7d1 (0x2aa7 bytes) 2024-04-24T00:35:23.311 INFO:teuthology.orchestra.run.smithi064.stdout:2754 trunc from 0x93923fd to 0xc04b6d9 2024-04-24T00:35:23.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:23.325 INFO:teuthology.orchestra.run.smithi064.stdout:2758 punch from 0x6ae43b6 to 0x6ae58f0, (0x153a bytes) 2024-04-24T00:35:23.348 INFO:teuthology.orchestra.run.smithi064.stdout:2759 trunc from 0xc04b6d9 to 0x16fe06b 2024-04-24T00:35:23.370 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.370 INFO:teuthology.orchestra.run.smithi064.stdout:2763 read 0x149b7dc thru 0x14a4ad5 (0x92fa bytes) 2024-04-24T00:35:23.371 INFO:teuthology.orchestra.run.smithi064.stdout:2764 read 0x21b209 thru 0x228ce8 (0xdae0 bytes) 2024-04-24T00:35:23.388 INFO:teuthology.orchestra.run.smithi064.stdout:2765 write 0x27638ae thru 0x27734e0 (0xfc33 bytes) 2024-04-24T00:35:23.392 INFO:teuthology.orchestra.run.smithi064.stdout:2766 read 0x1e80496 thru 0x1e8fa80 (0xf5eb bytes) 2024-04-24T00:35:23.393 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.393 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.395 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:23.389+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:23.419 INFO:teuthology.orchestra.run.smithi064.stdout:2770 write 0x8be9074 thru 0x8bebf47 (0x2ed4 bytes) 2024-04-24T00:35:23.425 INFO:teuthology.orchestra.run.smithi064.stdout:2772 punch from 0x674067b to 0x674589e, (0x5223 bytes) 2024-04-24T00:35:23.425 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.425 INFO:teuthology.orchestra.run.smithi064.stdout:2774 write 0x58e6213 thru 0x58e89ba (0x27a8 bytes) 2024-04-24T00:35:23.452 INFO:teuthology.orchestra.run.smithi064.stdout:2775 writesame 0xd047cc9 thru 0xd05743f data_size 0x3(0xf777 bytes) 2024-04-24T00:35:23.455 INFO:teuthology.orchestra.run.smithi064.stdout:2776 write 0x22ce95a thru 0x22dabb9 (0xc260 bytes) 2024-04-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.815102534999994, 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:35:23.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:35:23.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:35:23.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20552 3.9 e49/21 3:923662ec:::rbd_data.115ee4742374.000000000000000a:head v 49'88, mlcod=48'86)", 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:23.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.486408935, 2024-04-24T00:35:23.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062734230000000002, 2024-04-24T00:35:23.471 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973502+0000", 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.04e-06 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973677+0000", 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473200000000001 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973714+0000", 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7117999999999999e-05 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973734+0000", 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0102000000000002e-05 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973879+0000", 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145654 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973946+0000", 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6476999999999994e-05 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979687+0000", 2024-04-24T00:35:23.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057411140000000003 2024-04-24T00:35:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979773+0000", 2024-04-24T00:35:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6186000000000005e-05 2024-04-24T00:35:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.876827679000002, 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:23.476 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:35:23.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:35:23.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:35:23.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.482 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:23.483 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:23.483 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.876753823999998, 2024-04-24T00:35:23.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:35:23.483 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:23.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:23.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:23.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:23.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:35:23.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.484 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:23.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:35:23.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:35:23.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:35:23.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:35:23.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:35:23.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:35:23.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:35:23.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:35:23.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:35:23.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.489 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.490 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:35:23.490 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:23.490 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.180059040000003, 2024-04-24T00:35:23.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:35:23.490 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:35:23.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:35:23.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:35:23.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:35:23.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:35:23.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:35:23.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:35:23.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.496 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.374527080999997, 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:35:23.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:68687 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'253, mlcod=94'249)", 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.794515638, 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005827128, 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665394+0000", 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7700000000000004e-07 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665402+0000", 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4030000000000007e-06 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665414+0000", 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1467999999999999e-05 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665419+0000", 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1780000000000002e-06 2024-04-24T00:35:23.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665485+0000", 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5944000000000004e-05 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665507+0000", 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2359999999999999e-05 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.671176+0000", 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056689130000000003 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.671220+0000", 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4184999999999998e-05 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:69871 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'404, mlcod=94'396)", 2024-04-24T00:35:23.505 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:23.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6274383110000006, 2024-04-24T00:35:23.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055042449999999996, 2024-04-24T00:35:23.506 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.506 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.506 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.506 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832471+0000", 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9200000000000001e-07 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832475+0000", 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.873e-06 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832482+0000", 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832486+0000", 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2470000000000002e-06 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832537+0000", 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1233000000000002e-05 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832563+0000", 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5269000000000002e-05 2024-04-24T00:35:23.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.837951+0000", 2024-04-24T00:35:23.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053882720000000004 2024-04-24T00:35:23.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.837975+0000", 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3683e-05 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:69977 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'424, mlcod=94'419)", 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6076822669999995, 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056478099999999996, 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.511 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852227+0000", 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-04-24T00:35:23.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852231+0000", 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1489999999999996e-06 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852240+0000", 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0010000000000007e-06 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852245+0000", 2024-04-24T00:35:23.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.498e-06 2024-04-24T00:35:23.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852283+0000", 2024-04-24T00:35:23.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8195000000000003e-05 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852298+0000", 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4911e-05 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.857827+0000", 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055296210000000002 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.857874+0000", 2024-04-24T00:35:23.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6897999999999997e-05 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70740 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'523, mlcod=94'518)", 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4945286650000007, 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061709410000000001, 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965381+0000", 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.094e-06 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965389+0000", 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9580000000000007e-06 2024-04-24T00:35:23.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965423+0000", 2024-04-24T00:35:23.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3787e-05 2024-04-24T00:35:23.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965443+0000", 2024-04-24T00:35:23.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0390999999999998e-05 2024-04-24T00:35:23.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965583+0000", 2024-04-24T00:35:23.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013995500000000001 2024-04-24T00:35:23.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965606+0000", 2024-04-24T00:35:23.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3003999999999999e-05 2024-04-24T00:35:23.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971482+0000", 2024-04-24T00:35:23.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058757649999999998 2024-04-24T00:35:23.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971551+0000", 2024-04-24T00:35:23.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8986999999999995e-05 2024-04-24T00:35:23.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.523 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.524 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70743 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'524, mlcod=94'518)", 2024-04-24T00:35:23.524 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:23.524 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4938243369999995, 2024-04-24T00:35:23.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055099820000000001, 2024-04-24T00:35:23.525 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.525 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.525 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.525 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966086+0000", 2024-04-24T00:35:23.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1969999999999999e-06 2024-04-24T00:35:23.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966096+0000", 2024-04-24T00:35:23.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.995e-06 2024-04-24T00:35:23.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966113+0000", 2024-04-24T00:35:23.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7538e-05 2024-04-24T00:35:23.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966124+0000", 2024-04-24T00:35:23.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:35:23.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966181+0000", 2024-04-24T00:35:23.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6937000000000001e-05 2024-04-24T00:35:23.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966199+0000", 2024-04-24T00:35:23.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8002000000000001e-05 2024-04-24T00:35:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971556+0000", 2024-04-24T00:35:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053569029999999997 2024-04-24T00:35:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971594+0000", 2024-04-24T00:35:23.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8186999999999999e-05 2024-04-24T00:35:23.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.534 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70750 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'527, mlcod=94'519)", 2024-04-24T00:35:23.534 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:23.534 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4915477660000001, 2024-04-24T00:35:23.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005844307, 2024-04-24T00:35:23.535 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.535 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.535 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:23.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:23.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968362+0000", 2024-04-24T00:35:23.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4900000000000004e-07 2024-04-24T00:35:23.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968371+0000", 2024-04-24T00:35:23.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.619e-06 2024-04-24T00:35:23.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968388+0000", 2024-04-24T00:35:23.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7113999999999999e-05 2024-04-24T00:35:23.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968401+0000", 2024-04-24T00:35:23.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3495e-05 2024-04-24T00:35:23.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968434+0000", 2024-04-24T00:35:23.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3306000000000003e-05 2024-04-24T00:35:23.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968446+0000", 2024-04-24T00:35:23.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1668000000000001e-05 2024-04-24T00:35:23.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.974121+0000", 2024-04-24T00:35:23.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056745290000000002 2024-04-24T00:35:23.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.974205+0000", 2024-04-24T00:35:23.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4627000000000001e-05 2024-04-24T00:35:23.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.545 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70759 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'530, mlcod=94'521)", 2024-04-24T00:35:23.545 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:23.545 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4888369229999991, 2024-04-24T00:35:23.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054257760000000002, 2024-04-24T00:35:23.545 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.546 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.546 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:23.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971072+0000", 2024-04-24T00:35:23.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971073+0000", 2024-04-24T00:35:23.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4300000000000002e-07 2024-04-24T00:35:23.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971078+0000", 2024-04-24T00:35:23.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0220000000000004e-06 2024-04-24T00:35:23.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971086+0000", 2024-04-24T00:35:23.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4400000000000005e-06 2024-04-24T00:35:23.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971091+0000", 2024-04-24T00:35:23.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.763e-06 2024-04-24T00:35:23.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971138+0000", 2024-04-24T00:35:23.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6897000000000002e-05 2024-04-24T00:35:23.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971150+0000", 2024-04-24T00:35:23.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2480000000000001e-05 2024-04-24T00:35:23.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.976394+0000", 2024-04-24T00:35:23.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052434580000000003 2024-04-24T00:35:23.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.976498+0000", 2024-04-24T00:35:23.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103973 2024-04-24T00:35:23.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70804 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'539, mlcod=94'532)", 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4792495399999996, 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058417199999999999, 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:23.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980660+0000", 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.99e-07 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980665+0000", 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.95e-06 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980681+0000", 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6653000000000001e-05 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980686+0000", 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9420000000000003e-06 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980873+0000", 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018610100000000001 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980889+0000", 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6884e-05 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986449+0000", 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055591700000000004 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986501+0000", 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2321000000000003e-05 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70823 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'542, mlcod=94'536)", 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4766194460000008, 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057985090000000003, 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983290+0000", 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983296+0000", 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7969999999999997e-06 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983304+0000", 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4519999999999997e-06 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983309+0000", 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6809999999999997e-06 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983353+0000", 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3844999999999998e-05 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983365+0000", 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.235e-05 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.988997+0000", 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005632176 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.989088+0000", 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0587999999999994e-05 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70843 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'546, mlcod=94'537)", 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4738826140000008, 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059306039999999999, 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986027+0000", 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2e-07 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986031+0000", 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-04-24T00:35:23.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986039+0000", 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1880000000000002e-06 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986043+0000", 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2339999999999996e-06 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986133+0000", 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9783000000000004e-05 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986146+0000", 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2816000000000001e-05 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.991875+0000", 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057291429999999999 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.991957+0000", 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2027999999999995e-05 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70942 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'565, mlcod=94'560)", 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4553378860000006, 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005499884, 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004572+0000", 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.576e-06 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004583+0000", 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0383e-05 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004603+0000", 2024-04-24T00:35:23.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9738999999999999e-05 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004612+0000", 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6919999999999997e-06 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004691+0000", 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8771999999999997e-05 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004711+0000", 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0018e-05 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009995+0000", 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052835520000000004 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.010071+0000", 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6151999999999999e-05 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70944 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'567, mlcod=94'561)", 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4529833459999999, 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059596270000000003, 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:23.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006927+0000", 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006936+0000", 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6719999999999999e-06 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006960+0000", 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4249000000000001e-05 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006975+0000", 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4259e-05 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.007069+0000", 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4477999999999999e-05 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.007088+0000", 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8332999999999999e-05 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012838+0000", 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057502830000000001 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012885+0000", 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7206999999999997e-05 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70966 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'570, mlcod=94'563)", 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4506610930000008, 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059298320000000003, 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009249+0000", 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.155e-06 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009257+0000", 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.892e-06 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009275+0000", 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8272000000000001e-05 2024-04-24T00:35:23.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009288+0000", 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3107e-05 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009373+0000", 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4561999999999994e-05 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009407+0000", 2024-04-24T00:35:23.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4100000000000002e-05 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.015084+0000", 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056774750000000004 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.015177+0000", 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3269000000000004e-05 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70979 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'574, mlcod=94'565)", 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4476617879999996, 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054674650000000003, 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012248+0000", 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0920000000000001e-06 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012256+0000", 2024-04-24T00:35:23.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9119999999999998e-06 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012273+0000", 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7280000000000001e-05 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012295+0000", 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1866999999999999e-05 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:23.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012370+0000", 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5278999999999995e-05 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012409+0000", 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8226999999999998e-05 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.017649+0000", 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052406350000000004 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:23.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.017714+0000", 2024-04-24T00:35:23.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5172999999999994e-05 2024-04-24T00:35:23.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:23.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:23.585 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:23.586 INFO:teuthology.orchestra.run.smithi064.stdout:2777 write 0xa297ac2 thru 0xa299855 (0x1d94 bytes) 2024-04-24T00:35:23.586 INFO:teuthology.orchestra.run.smithi064.stdout:2778 read 0x10ebd51 thru 0x10fa561 (0xe811 bytes) 2024-04-24T00:35:23.586 INFO:teuthology.orchestra.run.smithi064.stdout:2779 trunc from 0xd057440 to 0xdba9b23 2024-04-24T00:35:23.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:23.587 INFO:teuthology.orchestra.run.smithi064.stdout:2780 write 0x6db241c thru 0x6dbb7c3 (0x93a8 bytes) 2024-04-24T00:35:23.587 INFO:teuthology.orchestra.run.smithi064.stdout:2781 writesame 0xbb3b7d6 thru 0xbb49068 data_size 0x1(0xd893 bytes) 2024-04-24T00:35:23.588 INFO:teuthology.orchestra.run.smithi064.stdout:2782 write 0xb94950b thru 0xb94fe75 (0x696b bytes) 2024-04-24T00:35:23.589 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:23.583+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:23.589 INFO:teuthology.orchestra.run.smithi064.stdout:2783 trunc from 0xdba9b23 to 0x85259d3 2024-04-24T00:35:23.604 INFO:teuthology.orchestra.run.smithi064.stdout:2785 punch from 0x226c1f to 0x232c9d, (0xc07e bytes) 2024-04-24T00:35:23.617 INFO:teuthology.orchestra.run.smithi064.stdout:2786 write 0x9f71159 thru 0x9f7625f (0x5107 bytes) 2024-04-24T00:35:23.623 INFO:teuthology.orchestra.run.smithi064.stdout:2788 read 0x24b4c4b thru 0x24c30ac (0xe462 bytes) 2024-04-24T00:35:23.631 INFO:teuthology.orchestra.run.smithi064.stdout:2789 trunc from 0x9f76260 to 0xe177a3c 2024-04-24T00:35:23.660 INFO:teuthology.orchestra.run.smithi064.stdout:2790 trunc from 0xe177a3c to 0x294a54b 2024-04-24T00:35:23.689 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:35:23.689 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:35:23.689 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 2, 3, 4, 6, 7, 5] 2024-04-24T00:35:23.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-04-24T00:35:23.690 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:23.687+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:23.691 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.709 INFO:teuthology.orchestra.run.smithi064.stdout:2792 trunc from 0x294a54b to 0xd58cbcc 2024-04-24T00:35:23.721 INFO:teuthology.orchestra.run.smithi064.stdout:2793 punch from 0x502c9 to 0x5d015, (0xcd4c bytes) 2024-04-24T00:35:23.721 INFO:teuthology.orchestra.run.smithi064.stdout:2794 punch from 0x1e6e425 to 0x1e7659c, (0x8177 bytes) 2024-04-24T00:35:23.721 INFO:teuthology.orchestra.run.smithi064.stdout:2795 write 0xca2f5ee thru 0xca3811b (0x8b2e bytes) 2024-04-24T00:35:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:23.731 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:23.731 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:23.752 INFO:teuthology.orchestra.run.smithi064.stdout:2796 trunc from 0xd58cbcc to 0x261082d 2024-04-24T00:35:23.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:23.801 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:23.797+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:23.803 INFO:teuthology.orchestra.run.smithi064.stdout:2797 writesame 0xcf7bd52 thru 0xcf884b9 data_size 0x18(0xc768 bytes) 2024-04-24T00:35:23.814 INFO:teuthology.orchestra.run.smithi064.stdout:2798 write 0xd60a6de thru 0xd61703e (0xc961 bytes) 2024-04-24T00:35:23.843 INFO:teuthology.orchestra.run.smithi064.stdout:2799 trunc from 0xd61703f to 0x132e347 2024-04-24T00:35:23.873 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.873 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.873 INFO:teuthology.orchestra.run.smithi064.stdout:2804 punch from 0x824c43 to 0x82a68f, (0x5a4c bytes) 2024-04-24T00:35:23.873 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.887 INFO:teuthology.orchestra.run.smithi064.stdout:2806 write 0x41fa8e8 thru 0x4204bbf (0xa2d8 bytes) 2024-04-24T00:35:23.891 INFO:teuthology.orchestra.run.smithi064.stdout:2807 read 0x15ae294 thru 0x15b28c0 (0x462d bytes) 2024-04-24T00:35:23.891 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:23.887+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:23.892 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:23.894 INFO:teuthology.orchestra.run.smithi064.stdout:2810 trunc from 0x4204bc0 to 0x552f3f6 2024-04-24T00:35:23.920 INFO:teuthology.orchestra.run.smithi064.stdout:2811 write 0x59b8f2f thru 0x59be545 (0x5617 bytes) 2024-04-24T00:35:23.922 INFO:teuthology.orchestra.run.smithi064.stdout:2812 read 0x53625be thru 0x5367358 (0x4d9b bytes) 2024-04-24T00:35:23.936 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:23.936 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:23.936 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:23.936 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:23.936 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:23.945 INFO:teuthology.orchestra.run.smithi064.stdout:2813 writesame 0xb63c678 thru 0xb647fc6 data_size 0x1b(0xb94f bytes) 2024-04-24T00:35:23.945 INFO:teuthology.orchestra.run.smithi064.stdout:2814 write 0xa6656a4 thru 0xa671f07 (0xc864 bytes) 2024-04-24T00:35:23.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:23.956 INFO:teuthology.orchestra.run.smithi064.stdout:2816 read 0x81cab6d thru 0x81ce823 (0x3cb7 bytes) 2024-04-24T00:35:23.957 INFO:teuthology.orchestra.run.smithi064.stdout:2818 read 0x3764b9c thru 0x37741d5 (0xf63a bytes) 2024-04-24T00:35:23.959 INFO:teuthology.orchestra.run.smithi064.stdout:2819 read 0xa1acda8 thru 0xa1af9c6 (0x2c1f bytes) 2024-04-24T00:35:23.959 INFO:teuthology.orchestra.run.smithi064.stdout:2821 read 0x32e8e42 thru 0x32ea2db (0x149a bytes) 2024-04-24T00:35:23.959 INFO:teuthology.orchestra.run.smithi064.stdout:2822 punch from 0xb5c0041 to 0xb5c4f06, (0x4ec5 bytes) 2024-04-24T00:35:23.959 INFO:teuthology.orchestra.run.smithi064.stdout:2823 read 0xb0d1c04 thru 0xb0e0fe5 (0xf3e2 bytes) 2024-04-24T00:35:23.959 INFO:teuthology.orchestra.run.smithi064.stdout:2825 read 0x408207c thru 0x408e417 (0xc39c bytes) 2024-04-24T00:35:23.963 INFO:teuthology.orchestra.run.smithi064.stdout:2826 trunc from 0xb647fc7 to 0xdcfe375 2024-04-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi064.stdout:2827 trunc from 0xdcfe375 to 0x2deefc1 2024-04-24T00:35:23.992 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:23.985+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:24.017 INFO:teuthology.orchestra.run.smithi064.stdout:2828 read 0x44b532 thru 0x44f7d6 (0x42a5 bytes) 2024-04-24T00:35:24.029 INFO:teuthology.orchestra.run.smithi064.stdout:2829 write 0x3e0f37c thru 0x3e1a985 (0xb60a bytes) 2024-04-24T00:35:24.034 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:24.034 INFO:teuthology.orchestra.run.smithi064.stdout:2833 punch from 0x1c36d44 to 0x1c43c37, (0xcef3 bytes) 2024-04-24T00:35:24.044 INFO:teuthology.orchestra.run.smithi064.stdout:2834 trunc from 0x3e1a986 to 0xa214b0e 2024-04-24T00:35:24.049 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:35:24.045+0000 7f841a5f8640 -1 mon.a@0(leader).osd e102 definitely_dead 0 2024-04-24T00:35:24.056 INFO:teuthology.orchestra.run.smithi064.stdout:2836 write 0x83a78ad thru 0x83b22df (0xaa33 bytes) 2024-04-24T00:35:24.080 INFO:teuthology.orchestra.run.smithi064.stdout:2838 write 0xdd72a99 thru 0xdd827cc (0xfd34 bytes) 2024-04-24T00:35:24.087 INFO:teuthology.orchestra.run.smithi064.stdout:2839 punch from 0xb265b53 to 0xb27078b, (0xac38 bytes) 2024-04-24T00:35:24.093 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:24.085+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2057 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2621440~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563678956000004, 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022810944999999999, 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2057 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528811+0000", 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528814+0000", 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.475e-06 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528820+0000", 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3860000000000004e-06 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528825+0000", 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551449+0000", 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022624293 2024-04-24T00:35:24.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551526+0000", 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6278000000000004e-05 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551622+0000", 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6162000000000005e-05 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563655041999994, 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:35:24.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:35:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563627867999998, 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.109 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563601762999994, 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.110 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:24.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:35:24.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563190782000007, 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:35:24.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563170972999998, 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.119 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:35:24.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.123 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563117914000003, 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:35:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563083058000004, 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563061493999996, 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.563032002, 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:35:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.56301114, 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:35:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.562988566000001, 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:35:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:35:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.880729067999994, 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.880378152999995, 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:35:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:35:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.873489832999994, 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:35:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44653 3.c e79/56 3:3545ecd3:::rbd_data.115ec7ed39de.0000000000000002:head v 79'268, mlcod=77'266)", 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.202746112, 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022668765, 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.889744+0000", 2024-04-24T00:35:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7800000000000002e-07 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892860+0000", 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003115821 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892903+0000", 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3176999999999997e-05 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.892926+0000", 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2459999999999998e-05 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.893026+0000", 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010062 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.893059+0000", 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2806999999999999e-05 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.912296+0000", 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019237431999999999 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.912412+0000", 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011597 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44656 3.7 e79/44 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:head v 79'103, mlcod=77'101)", 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.198163366999999, 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026507441999999999, 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894327+0000", 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1e-07 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898194+0000", 2024-04-24T00:35:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038672889999999999 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898203+0000", 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6340000000000007e-06 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898208+0000", 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2009999999999998e-06 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898262+0000", 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4466999999999999e-05 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898280+0000", 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7492000000000001e-05 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920764+0000", 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022483974 2024-04-24T00:35:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920834+0000", 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0174999999999998e-05 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44661 3.17 e79/59 3:eff1c653:::rbd_data.115ec7ed39de.0000000000000005:head v 79'91, mlcod=77'89)", 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.196296078, 2024-04-24T00:35:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024065629000000002, 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8299999999999996e-07 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897771+0000", 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015761690000000001 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897786+0000", 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5051999999999999e-05 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897794+0000", 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4689999999999997e-06 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897869+0000", 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.47e-05 2024-04-24T00:35:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897899+0000", 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9777e-05 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920181+0000", 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022282560999999999 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920259+0000", 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8218000000000002e-05 2024-04-24T00:35:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.615208206999998, 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:35:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.604931790999998, 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:35:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:24.190 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:24.190 INFO:teuthology.orchestra.run.smithi064.stdout:2840 trunc from 0xdd827cd to 0x365952e 2024-04-24T00:35:24.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:24.192 INFO:teuthology.orchestra.run.smithi064.stdout:2841 punch from 0x2ee2517 to 0x2ef0487, (0xdf70 bytes) 2024-04-24T00:35:24.192 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:24.192 INFO:teuthology.orchestra.run.smithi064.stdout:2844 writesame 0x84feb25 thru 0x850a3d5 data_size 0x1(0xb8b1 bytes) 2024-04-24T00:35:24.192 INFO:teuthology.orchestra.run.smithi064.stdout:2846 write 0xdd577d0 thru 0xdd5f058 (0x7889 bytes) 2024-04-24T00:35:24.192 INFO:teuthology.orchestra.run.smithi064.stdout:2847 write 0x804053c thru 0x8049337 (0x8dfc bytes) 2024-04-24T00:35:24.193 INFO:teuthology.orchestra.run.smithi064.stdout:2848 writesame 0xd2a805d thru 0xd2b2b91 data_size 0x1(0xab35 bytes) 2024-04-24T00:35:24.194 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:24.191+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:24.295 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:24.289+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:24.328 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:24.328 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:24.329 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:24.329 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:24.336 INFO:teuthology.orchestra.run.smithi064.stdout:2849 trunc from 0xdd5f059 to 0x975b7a 2024-04-24T00:35:24.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:24.364 INFO:teuthology.orchestra.run.smithi064.stdout:2851 read 0x18050f thru 0x1890e6 (0x8bd8 bytes) 2024-04-24T00:35:24.365 INFO:teuthology.orchestra.run.smithi064.stdout:2852 read 0x3ae297 thru 0x3b654e (0x82b8 bytes) 2024-04-24T00:35:24.366 INFO:teuthology.orchestra.run.smithi064.stdout:2853 punch from 0x48c049 to 0x492503, (0x64ba bytes) 2024-04-24T00:35:24.395 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:24.389+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:24.401 INFO:teuthology.orchestra.run.smithi064.stdout:2854 write 0x9482f53 thru 0x94879a7 (0x4a55 bytes) 2024-04-24T00:35:24.422 INFO:teuthology.orchestra.run.smithi064.stdout:2855 trunc from 0x94879a8 to 0x43da0fb 2024-04-24T00:35:24.455 INFO:teuthology.orchestra.run.smithi064.stdout:2857 punch from 0x1b0259a to 0x1b0c4c3, (0x9f29 bytes) 2024-04-24T00:35:24.472 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:24.472 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:24.472 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:24.473 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:24.473 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:24.480 INFO:teuthology.orchestra.run.smithi064.stdout:2858 write 0x8412110 thru 0x8415b0e (0x39ff bytes) 2024-04-24T00:35:24.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:24.488 INFO:teuthology.orchestra.run.smithi064.stdout:2859 write 0x46ad840 thru 0x46b3066 (0x5827 bytes) 2024-04-24T00:35:24.491 INFO:teuthology.orchestra.run.smithi064.stdout:2860 read 0x6708c99 thru 0x6714fe6 (0xc34e bytes) 2024-04-24T00:35:24.492 INFO:teuthology.orchestra.run.smithi064.stdout:2862 read 0x7a2cbd thru 0x7b03e8 (0xd72c bytes) 2024-04-24T00:35:24.494 INFO:teuthology.orchestra.run.smithi064.stdout:2864 punch from 0x35d4d46 to 0x35e1517, (0xc7d1 bytes) 2024-04-24T00:35:24.494 INFO:teuthology.orchestra.run.smithi064.stdout:2867 read 0x4ac677e thru 0x4ad2d36 (0xc5b9 bytes) 2024-04-24T00:35:24.496 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:24.496 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:24.489+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:24.496 INFO:teuthology.orchestra.run.smithi064.stdout:2870 write 0x32f9cf1 thru 0x32fe548 (0x4858 bytes) 2024-04-24T00:35:24.500 INFO:teuthology.orchestra.run.smithi064.stdout:2871 write 0x697dbb7 thru 0x698bc0f (0xe059 bytes) 2024-04-24T00:35:24.534 INFO:teuthology.orchestra.run.smithi064.stdout:2872 write 0xed702d2 thru 0xed76336 (0x6065 bytes) 2024-04-24T00:35:24.542 INFO:teuthology.orchestra.run.smithi064.stdout:2874 read 0xe38c358 thru 0xe38d7a2 (0x144b bytes) 2024-04-24T00:35:24.542 INFO:teuthology.orchestra.run.smithi064.stdout:2875 write 0xe4fde8e thru 0xe50a837 (0xc9aa bytes) 2024-04-24T00:35:24.552 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:24.588 INFO:teuthology.orchestra.run.smithi064.stdout:2879 trunc from 0xed76337 to 0x1b232f1 2024-04-24T00:35:24.628 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:24.628 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:24.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:24.628 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:24.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.628 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.976092679999994, 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:35:24.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:35:24.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:35:24.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:35:24.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:35:24.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:35:24.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:35:24.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:35:24.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:35:24.636 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.636 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.636 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.636 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:24.636 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:24.636 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.976016481000002, 2024-04-24T00:35:24.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:35:24.637 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.637 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.637 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.637 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.637 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.637 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.638 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:35:24.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.638 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:24.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:24.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:35:24.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:35:24.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:35:24.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:35:24.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:35:24.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:35:24.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:35:24.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:35:24.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:35:24.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:35:24.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:35:24.643 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.644 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.644 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.644 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:24.644 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:24.644 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.975558307, 2024-04-24T00:35:24.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.645 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:24.650 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.975500999999994, 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:35:24.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:35:24.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:35:24.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.975242034000004, 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.657 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.658 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:35:24.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.658 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:24.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:24.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:35:24.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:35:24.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:35:24.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:35:24.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:35:24.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:35:24.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:35:24.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:35:24.663 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.974320562000003, 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.664 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:35:24.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:35:24.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.402505832000003, 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:24.668 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:24.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:24.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.386976583000006, 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:24.674 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:24.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:24.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.758253263, 2024-04-24T00:35:24.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:24.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.743218135999996, 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.684 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.980720138000002, 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.688 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:24.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.966897179999997, 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:24.693 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:24.697 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.739755995000003, 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:24.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:24.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.703 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.705 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:24.705 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:24.705 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.727966344999999, 2024-04-24T00:35:24.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:24.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.709 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30704 3.1a 3:5e74e97a:::rbd_data.115e8210e51a.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.364474426000001, 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043351496000000003, 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30704 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252358+0000", 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1790000000000001e-06 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252364+0000", 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6549999999999998e-06 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252381+0000", 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6767000000000001e-05 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252390+0000", 2024-04-24T00:35:24.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9959999999999999e-06 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252492+0000", 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102563 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252668+0000", 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017553499999999999 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:24.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.254545+0000", 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001877037 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295422+0000", 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040876527000000003 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295513+0000", 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1838999999999999e-05 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295614+0000", 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010027900000000001 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295708+0000", 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4118999999999998e-05 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:24.715 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.733926281000002, 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:24.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33518 3.b e70/59 ondisk, result = 0)", 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.722602752999997, 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042187448000000002, 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.721 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894230+0000", 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7680000000000001e-06 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894239+0000", 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3319999999999998e-06 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894266+0000", 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049000000000001e-05 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894281+0000", 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5245000000000001e-05 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936247+0000", 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041965833000000001 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936257+0000", 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936415+0000", 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158605 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33734 3.b 3:d4ebe596:::rbd_data.115ec4f9b3a7.000000000000000e:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.097560557999998, 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041668288999999997, 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.725 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33734 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519272+0000", 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9870000000000002e-06 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519282+0000", 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0203e-05 2024-04-24T00:35:24.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519310+0000", 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7880000000000001e-05 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519324+0000", 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519402+0000", 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7865000000000001e-05 2024-04-24T00:35:24.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519481+0000", 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9103000000000005e-05 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.522670+0000", 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003189 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560812+0000", 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038141427999999998 2024-04-24T00:35:24.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560831+0000", 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9820999999999999e-05 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560886+0000", 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5087999999999998e-05 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560938+0000", 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1993999999999997e-05 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44651 3.6 e79/59 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head v 79'72, mlcod=77'70)", 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.727853190000001, 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048315631999999997, 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.731 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:24.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892459+0000", 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034802259999999999 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892475+0000", 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5998000000000002e-05 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892485+0000", 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8679999999999994e-06 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892653+0000", 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016800800000000001 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892721+0000", 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8953000000000002e-05 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937243+0000", 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044521666000000001 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937293+0000", 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0043000000000002e-05 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.735 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44659 3.1d e79/53 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head v 79'105, mlcod=77'103)", 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.720372007000002, 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038358476000000002, 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896460+0000", 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2399999999999997e-07 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897721+0000", 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012617049999999999 2024-04-24T00:35:24.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897744+0000", 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2982000000000001e-05 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897756+0000", 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1559999999999999e-05 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:24.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897836+0000", 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0434999999999998e-05 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897871+0000", 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5147000000000001e-05 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934696+0000", 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036824511999999997 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:24.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934817+0000", 2024-04-24T00:35:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121311 2024-04-24T00:35:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:24.740 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:24.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:35:24.741 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:24.621+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:24.743 INFO:teuthology.orchestra.run.smithi064.stdout:2880 write 0xa33de04 thru 0xa3423da (0x45d7 bytes) 2024-04-24T00:35:24.743 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:24.743 INFO:teuthology.orchestra.run.smithi064.stdout:2882 read 0x5c13890 thru 0x5c1d154 (0x98c5 bytes) 2024-04-24T00:35:24.743 INFO:teuthology.orchestra.run.smithi064.stdout:2883 write 0x294d05b thru 0x2950600 (0x35a6 bytes) 2024-04-24T00:35:24.743 INFO:teuthology.orchestra.run.smithi064.stdout:2884 punch from 0x360760a to 0x360f816, (0x820c bytes) 2024-04-24T00:35:24.743 INFO:teuthology.orchestra.run.smithi064.stdout:2886 write 0x65849a6 thru 0x659087e (0xbed9 bytes) 2024-04-24T00:35:24.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:24.744 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:24.739+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:24.744 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.7. 2024-04-24T00:35:24.744 INFO:teuthology.orchestra.run.smithi064.stdout:2888 punch from 0x1200142 to 0x120d529, (0xd3e7 bytes) 2024-04-24T00:35:24.745 INFO:teuthology.orchestra.run.smithi064.stdout:2893 write 0xc0a5c33 thru 0xc0ad6d5 (0x7aa3 bytes) 2024-04-24T00:35:24.745 INFO:teuthology.orchestra.run.smithi064.stdout:2894 write 0xa65d2e1 thru 0xa66a094 (0xcdb4 bytes) 2024-04-24T00:35:24.746 INFO:teuthology.orchestra.run.smithi064.stdout:2895 read 0x1010815 thru 0x101889a (0x8086 bytes) 2024-04-24T00:35:24.746 INFO:teuthology.orchestra.run.smithi064.stdout:2896 writesame 0x2b4d58a thru 0x2b591e1 data_size 0xe(0xbc58 bytes) 2024-04-24T00:35:24.746 INFO:teuthology.orchestra.run.smithi064.stdout:2898 punch from 0xa51782 to 0xa53674, (0x1ef2 bytes) 2024-04-24T00:35:24.746 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:24.746 INFO:teuthology.orchestra.run.smithi064.stdout:2900 read 0x9cc6a1f thru 0x9cd4e8e (0xe470 bytes) 2024-04-24T00:35:24.746 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:24.747 INFO:teuthology.orchestra.run.smithi064.stdout:2902 trunc from 0xc0ad6d6 to 0xa7d398 2024-04-24T00:35:24.842 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:24.837+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:24.915 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:24.916 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:24.916 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:24.916 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:24.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:24.943 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:24.937+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:25.044 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:25.037+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:25.052 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:25.052 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:25.052 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:25.052 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:25.052 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:25.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:25.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:25.144 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:25.139+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.472994053000001, 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:35:25.187 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:35:25.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:25.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.472896128000002, 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:35:25.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:35:25.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 44.113516595999997, 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.199 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:35:25.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:35:25.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30133 3.1f e66/59 ondisk, result = 0)", 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 44.100923049000002, 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060466079999999998, 2024-04-24T00:35:25.205 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080440+0000", 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080446+0000", 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6699999999999999e-06 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080465+0000", 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080476+0000", 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0937999999999999e-05 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140869+0000", 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060392759999999997 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140882+0000", 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3131e-05 2024-04-24T00:35:25.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140905+0000", 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2642000000000001e-05 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30705 3.2 3:45c09e0c:::rbd_data.115e8210e51a.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.928910135000002, 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063528001000000001, 2024-04-24T00:35:25.209 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30705 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252455+0000", 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.303e-06 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252477+0000", 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1970000000000001e-05 2024-04-24T00:35:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252515+0000", 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7616e-05 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252536+0000", 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376999999999999e-05 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252696+0000", 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160192 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252945+0000", 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024906400000000001 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.256753+0000", 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038072129999999998 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315658+0000", 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058905258000000002 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315790+0000", 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013223999999999999 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315894+0000", 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103886 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315980+0000", 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5882000000000002e-05 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30705 3.2 e66/53 ondisk, result = 0)", 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.913698031999999, 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048330598000000002, 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267667+0000", 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.96e-06 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267683+0000", 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6336000000000001e-05 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267730+0000", 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7079000000000001e-05 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267757+0000", 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6364e-05 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315724+0000", 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047967018 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315742+0000", 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7935999999999999e-05 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315995+0000", 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025290499999999999 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33506 3.c 3:3787ff34:::rbd_data.115ec4f9b3a7.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 39.383313991000001, 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061037313000000003, 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.220 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33506 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798052+0000", 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798067+0000", 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5160000000000001e-05 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798107+0000", 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9946e-05 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798130+0000", 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2637e-05 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798238+0000", 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010841299999999999 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798344+0000", 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010631199999999999 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.801407+0000", 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030623600000000001 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858616+0000", 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057209342000000003 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858879+0000", 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000262952 2024-04-24T00:35:25.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858992+0000", 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113429 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859085+0000", 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2925999999999998e-05 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.225 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33506 3.c e70/56 ondisk, result = 0)", 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 39.373681224000002, 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051419185999999999, 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807682+0000", 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807687+0000", 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807704+0000", 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7424000000000001e-05 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807714+0000", 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3400000000000004e-06 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858714+0000", 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051000248999999998 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858732+0000", 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7995999999999998e-05 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859100+0000", 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036787400000000002 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.229 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35694 3.1b 3:dcf0eb61:::rbd_data.115ec5833ea9.00000000000000aa:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.261865372999999, 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049399446, 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35694 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919498+0000", 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919504+0000", 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919520+0000", 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5908000000000001e-05 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919532+0000", 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2964e-05 2024-04-24T00:35:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919630+0000", 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7096999999999995e-05 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919734+0000", 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104128 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.922056+0000", 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023219149999999999 2024-04-24T00:35:25.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968775+0000", 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046719274999999998 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968817+0000", 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-05 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968858+0000", 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1440999999999998e-05 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968896+0000", 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7741000000000003e-05 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:25.235 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.290532609, 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:25.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:35:25.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:35:25.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:35:25.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44664 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 524288~131072 [fadvise_sequential+fadvise_nocache],write 1179648~1048576 [fadvise_sequential+fadvise_nocache],write 10616832~2048 [fadvise_sequential+fadvise_nocache],write 10627072~4096 [fadvise_sequential+fadvise_nocache],write 10639360~4096 [fadvise_sequential+fadvise_nocache],write 10651648~4096 [fadvise_sequential+fadvise_nocache],write 10663936~4096 [fadvise_sequential+fadvise_nocache],write 10676224~4096 [fadvise_sequential+fadvise_nocache],write 10688512~4096 [fadvise_sequential+fadvise_nocache],write 10700800~4096 [fadvise_sequential+fadvise_nocache],write 10713088~4096 [fadvise_sequential+fadvise_nocache],write 10725376~4096 [fadvise_sequential+fadvise_nocache],write 10737664~4096 [fadvise_sequential+fadvise_nocache],write 10754048~4096 [fadvise_sequential+fadvise_nocache],write 10766336~4096 [fadvise_sequential+fadvise_nocache],write 10778624~4096 [fadvise_sequential+fadvise_nocache],write 10790912~4096 [fadvise_sequential+fadvise_nocache],write 10803200~4096 [fadvise_sequential+fadvise_nocache],write 10815488~4096 [fadvise_sequential+fadvise_nocache],write 10827776~4096 [fadvise_sequential+fadvise_nocache],write 10840064~4096 [fadvise_sequential+fadvise_nocache],write 10852352~4096 [fadvise_sequential+fadvise_nocache],write 10864640~4096 [fadvise_sequential+fadvise_nocache],write 10876928~2048 [fadvise_sequential+fadvise_nocache],write 10881024~4096 [fadvise_sequential+fadvise_nocache],write 10893312~4096 [fadvise_sequential+fadvise_nocache],write 10905600~4096 [fadvise_sequential+fadvise_nocache],write 10917888~4096 [fadvise_sequential+fadvise_nocache],write 10930176~4096 [fadvise_sequential+fadvise_nocache],write 10942464~4096 [fadvise_sequential+fadvise_nocache],write 10954752~4096 [fadvise_sequential+fadvise_nocache],write 10967040~4096 [fadvise_sequential+fadvise_nocache],write 10979328~4096 [fadvise_sequential+fadvise_nocache],write 10991616~4096 [fadvise_sequential+fadvise_nocache],write 11003904~4096 [fadvise_sequential+fadvise_nocache],write 11010048~2048 [fadvise_sequential+fadvise_nocache],write 11020288~4096 [fadvise_sequential+fadvise_nocache],write 11032576~4096 [fadvise_sequential+fadvise_nocache],write 11044864~4096 [fadvise_sequential+fadvise_nocache],write 11057152~4096 [fadvise_sequential+fadvise_nocache],write 11069440~4096 [fadvise_sequential+fadvise_nocache],write 11081728~4096 [fadvise_sequential+fadvise_nocache],write 11094016~4096 [fadvise_sequential+fadvise_nocache],write 11106304~4096 [fadvise_sequential+fadvise_nocache],write 11118592~4096 [fadvise_sequential+fadvise_nocache],write 11130880~4096 [fadvise_sequential+fadvise_nocache],write 14417920~4096 [fadvise_sequential+fadvise_nocache],write 14426112~8192 [fadvise_sequential+fadvise_nocache],write 14438400~8192 [fadvise_sequential+fadvise_nocache],write 14450688~8192 [fadvise_sequential+fadvise_nocache],write 14462976~8192 [fadvise_sequential+fadvise_nocache],write 14475264~8192 [fadvise_sequential+fadvise_nocache],write 14487552~8192 [fadvise_sequential+fadvise_nocache],write 14499840~8192 [fadvise_sequential+fadvise_nocache],write 14512128~8192 [fadvise_sequential+fadvise_nocache],write 14524416~8192 [fadvise_sequential+fadvise_nocache],write 14536704~8192 [fadvise_sequential+fadvise_nocache],write 14553088~8192 [fadvise_sequential+fadvise_nocache],write 14565376~8192 [fadvise_sequential+fadvise_nocache],write 14577664~8192 [fadvise_sequential+fadvise_nocache],write 14589952~8192 [fadvise_sequential+fadvise_nocache],write 14602240~8192 [fadvise_sequential+fadvise_nocache],write 14614528~8192 [fadvise_sequential+fadvise_nocache],write 14626816~8192 [fadvise_sequential+fadvise_nocache],write 14639104~8192 [fadvise_sequential+fadvise_nocache],write 14651392~8192 [fadvise_sequential+fadvise_nocache],write 14663680~8192 [fadvise_sequential+fadvise_nocache],write 14675968~4096 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.278459288000001, 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768734, 2024-04-24T00:35:25.241 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44664 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902904+0000", 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5179999999999999e-06 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.904804+0000", 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018996390000000001 2024-04-24T00:35:25.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905393+0000", 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058872600000000003 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905408+0000", 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5073e-05 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905482+0000", 2024-04-24T00:35:25.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4284999999999998e-05 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905785+0000", 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030266100000000002 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.907249+0000", 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001464899 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.932396+0000", 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025146379 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958308+0000", 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025912418999999999 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958527+0000", 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218297 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958671+0000", 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144838 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46463 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:1d [read 360448~8192 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.583282611000001, 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042258337999999999, 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.247 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.249 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.249 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46463 2024-04-24T00:35:25.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.249 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598080+0000", 2024-04-24T00:35:25.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598083+0000", 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6740000000000001e-06 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598091+0000", 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1089999999999998e-06 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598097+0000", 2024-04-24T00:35:25.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5370000000000003e-06 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.635329+0000", 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037231775000000002 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.635418+0000", 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8629999999999997e-05 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640338+0000", 2024-04-24T00:35:25.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049199730000000002 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46464 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.583256687999999, 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042285080000000003, 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46464 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598106+0000", 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7700000000000003e-07 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598109+0000", 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8080000000000001e-06 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598117+0000", 2024-04-24T00:35:25.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2430000000000005e-06 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598124+0000", 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6000000000000003e-06 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640379+0000", 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042255297999999997 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640390+0000", 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1253999999999999e-05 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.702772469999999, 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:35:25.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.677503863999998, 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.266 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:67305 3.17 e89/59 ondisk, result = 0)", 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 16.326729788000002, 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049974565999999998, 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854633+0000", 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854637+0000", 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.946e-06 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854649+0000", 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2507000000000001e-05 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854658+0000", 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6330000000000002e-06 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904468+0000", 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049809535000000002 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904479+0000", 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-05 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904607+0000", 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012738300000000001 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.270 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.8243001440000004, 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.271 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:35:25.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:35:25.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.7916950299999996, 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:35:25.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.3871203359999997, 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:35:25.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:35:25.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:25.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:25.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.3292801059999997, 2024-04-24T00:35:25.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:25.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:35:25.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:35:25.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:25.290 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:25.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:25.292 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:25.287+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:25.294 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:25.294 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:25.310 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:25.311 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:25.327 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:25.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:25.343 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:25.343 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:25.364 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:25.382 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:25.382 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:25.393 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:25.387+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:25.402 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:25.402 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:25.423 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:25.423 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:25.462 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:25.462 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:25.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:25.518 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:25.515+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:25.594 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:25.589+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:25.641 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:25.642 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:25.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:25.689 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:35:25.695 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:25.689+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:25.782 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:25.782 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:25.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:25.796 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:25.791+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:25.896 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:25.889+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:25.960 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:25.960 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:25.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:25.997 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:25.989+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:26.089 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:26.089 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:26.098 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:26.095+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:26.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:26.199 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:26.195+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:26.271 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:26.271 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:26.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:26.326 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:26.323+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:26.401 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:26.397+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:26.421 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:26.421 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:26.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:26.502 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:26.499+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:26.603 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:26.597+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:26.606 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:26.606 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:26.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:26.706 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:26.701+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:26.791 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:26.791 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:26.804 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:26.799+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:26.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:26.906 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:26.903+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:26.978 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:26.978 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:26.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:27.007 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:27.003+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:27.108 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:27.103+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:27.127 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:27.127 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:27.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:27.184 INFO:teuthology.orchestra.run.smithi064.stdout:2903 read 0x56fe69 thru 0x57b6c8 (0xb860 bytes) 2024-04-24T00:35:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:2904 read 0x79b1bf thru 0x79fa44 (0x4886 bytes) 2024-04-24T00:35:27.209 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:27.201+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:27.213 INFO:teuthology.orchestra.run.smithi064.stdout:2905 writesame 0xa1628c3 thru 0xa16a3a7 data_size 0x1(0x7ae5 bytes) 2024-04-24T00:35:27.213 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:27.213 INFO:teuthology.orchestra.run.smithi064.stdout:2908 write 0xc965f2 thru 0xca0f59 (0xa968 bytes) 2024-04-24T00:35:27.219 INFO:teuthology.orchestra.run.smithi064.stdout:2909 read 0x90651f thru 0x90eb13 (0x85f5 bytes) 2024-04-24T00:35:27.220 INFO:teuthology.orchestra.run.smithi064.stdout:2910 write 0x9f54cce thru 0x9f5d1ef (0x8522 bytes) 2024-04-24T00:35:27.228 INFO:teuthology.orchestra.run.smithi064.stdout:2911 write 0x6de4e11 thru 0x6df38f7 (0xeae7 bytes) 2024-04-24T00:35:27.234 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:27.234 INFO:teuthology.orchestra.run.smithi064.stdout:2913 punch from 0x26d8a58 to 0x26e01fc, (0x77a4 bytes) 2024-04-24T00:35:27.234 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:27.234 INFO:teuthology.orchestra.run.smithi064.stdout:2915 punch from 0x5ae9fa to 0x5bb3c0, (0xc9c6 bytes) 2024-04-24T00:35:27.244 INFO:teuthology.orchestra.run.smithi064.stdout:2917 write 0xa94c234 thru 0xa958c06 (0xc9d3 bytes) 2024-04-24T00:35:27.256 INFO:teuthology.orchestra.run.smithi064.stdout:2918 trunc from 0xa958c07 to 0x9fa0e14 2024-04-24T00:35:27.270 INFO:teuthology.orchestra.run.smithi064.stdout:2919 write 0x3c137e1 thru 0x3c1a797 (0x6fb7 bytes) 2024-04-24T00:35:27.280 INFO:teuthology.orchestra.run.smithi064.stdout:2920 trunc from 0x9fa0e14 to 0xd96a5b5 2024-04-24T00:35:27.292 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:27.292 INFO:teuthology.orchestra.run.smithi064.stdout:2922 writesame 0x87123ee thru 0x8721da6 data_size 0x1(0xf9b9 bytes) 2024-04-24T00:35:27.292 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:27.292 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:27.310 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:27.310 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:27.303+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:27.410 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:27.405+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:27.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi064.stdout:2923 write 0xead26d1 thru 0xeae1903 (0xf233 bytes) 2024-04-24T00:35:27.474 INFO:teuthology.orchestra.run.smithi064.stdout:2924 trunc from 0xeae1904 to 0xb402aaf 2024-04-24T00:35:27.490 INFO:teuthology.orchestra.run.smithi064.stdout:2925 read 0x99fe03f thru 0x9a08aa4 (0xaa66 bytes) 2024-04-24T00:35:27.490 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:27.491 INFO:teuthology.orchestra.run.smithi064.stdout:2928 read 0x62bbdea thru 0x62cadfa (0xf011 bytes) 2024-04-24T00:35:27.492 INFO:teuthology.orchestra.run.smithi064.stdout:2929 write 0x4deb28e thru 0x4dec6a5 (0x1418 bytes) 2024-04-24T00:35:27.509 INFO:teuthology.orchestra.run.smithi064.stdout:2930 writesame 0xc5f37ce thru 0xc5fee5e data_size 0x1b(0xb691 bytes) 2024-04-24T00:35:27.510 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:27.511 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:27.511 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:27.505+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:27.513 INFO:teuthology.orchestra.run.smithi064.stdout:2932 trunc from 0xc5fee5f to 0xb49fee8 2024-04-24T00:35:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:2934 write 0xf11b3b thru 0xf1a5b1 (0x8a77 bytes) 2024-04-24T00:35:27.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:27.537 INFO:teuthology.orchestra.run.smithi064.stdout:2935 trunc from 0xb49fee8 to 0xd6d7a4f 2024-04-24T00:35:27.549 INFO:teuthology.orchestra.run.smithi064.stdout:2936 read 0xd396e30 thru 0xd398db8 (0x1f89 bytes) 2024-04-24T00:35:27.560 INFO:teuthology.orchestra.run.smithi064.stdout:2937 writesame 0xe0a959b thru 0xe0a99a9 data_size 0x1(0x40f bytes) 2024-04-24T00:35:27.573 INFO:teuthology.orchestra.run.smithi064.stdout:2938 writesame 0xce23b81 thru 0xce2d55d data_size 0x11(0x99dd bytes) 2024-04-24T00:35:27.585 INFO:teuthology.orchestra.run.smithi064.stdout:2939 write 0x7ebee3a thru 0x7ec8f1d (0xa0e4 bytes) 2024-04-24T00:35:27.595 INFO:teuthology.orchestra.run.smithi064.stdout:2940 read 0x9ff27ae thru 0x9ff8349 (0x5b9c bytes) 2024-04-24T00:35:27.595 INFO:teuthology.orchestra.run.smithi064.stdout:2944 read 0x4c3896c thru 0x4c394ac (0xb41 bytes) 2024-04-24T00:35:27.596 INFO:teuthology.orchestra.run.smithi064.stdout:2945 punch from 0xaf7ec3 to 0xaff364, (0x74a1 bytes) 2024-04-24T00:35:27.596 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:27.596 INFO:teuthology.orchestra.run.smithi064.stdout:2948 read 0x40a2703 thru 0x40aa98f (0x828d bytes) 2024-04-24T00:35:27.597 INFO:teuthology.orchestra.run.smithi064.stdout:2949 write 0xca46346 thru 0xca4d8ca (0x7585 bytes) 2024-04-24T00:35:27.612 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:27.607+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:27.618 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:27.618 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:27.626 INFO:teuthology.orchestra.run.smithi064.stdout:2952 trunc from 0xe0a99aa to 0x2db7eb 2024-04-24T00:35:27.641 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:27.641 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:2954 punch from 0x2b593b to 0x2b841d, (0x2ae2 bytes) 2024-04-24T00:35:27.662 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:27.663 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:27.670 INFO:teuthology.orchestra.run.smithi064.stdout:2955 trunc from 0x2db7eb to 0xdcd8ef1 2024-04-24T00:35:27.679 INFO:teuthology.orchestra.run.smithi064.stdout:2956 punch from 0x21b907b to 0x21c6a23, (0xd9a8 bytes) 2024-04-24T00:35:27.679 INFO:teuthology.orchestra.run.smithi064.stdout:2958 write 0x7365675 thru 0x7366348 (0xcd4 bytes) 2024-04-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi064.stdout:2959 read 0xb312bd7 thru 0xb314884 (0x1cae bytes) 2024-04-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi064.stdout:2960 write 0x2587126 thru 0x2588855 (0x1730 bytes) 2024-04-24T00:35:27.683 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:27.683 INFO:teuthology.orchestra.run.smithi064.stdout:2964 read 0xd03ecdd thru 0xd04b03a (0xc35e bytes) 2024-04-24T00:35:27.683 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:27.683 INFO:teuthology.orchestra.run.smithi064.stdout:2968 writesame 0x8afd07c thru 0x8b07a8f data_size 0x23(0xaa14 bytes) 2024-04-24T00:35:27.684 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:27.684 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:27.689 INFO:teuthology.orchestra.run.smithi064.stdout:2970 writesame 0x52d35f6 thru 0x52ddf1a data_size 0x2b(0xa925 bytes) 2024-04-24T00:35:27.691 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:27.691 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:27.693 INFO:teuthology.orchestra.run.smithi064.stdout:2971 writesame 0x22886c0 thru 0x2291649 data_size 0x2(0x8f8a bytes) 2024-04-24T00:35:27.713 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:27.707+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:27.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:27.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:35:27.716 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:27.716 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:27.738 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:27.738 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:27.759 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:27.759 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:27.796 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:27.796 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:27.813 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:27.807+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:27.866 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:27.866 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:27.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:27.866 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:27.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.866 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:5260 3.4 e36/21 3:24775dcd:::rbd_data.115ed6c33904.0000000000000027:head v 36'24, mlcod=34'22)", 2024-04-24T00:35:27.866 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.218539824999993, 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058991160000000003, 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644806+0000", 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644807+0000", 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644875+0000", 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7787000000000007e-05 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644892+0000", 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7458000000000001e-05 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644901+0000", 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3179999999999996e-06 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.644999+0000", 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7658000000000006e-05 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.645052+0000", 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3390000000000002e-05 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650635+0000", 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055822809999999997 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:08.650705+0000", 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0635000000000004e-05 2024-04-24T00:35:27.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:20552 3.9 e49/21 3:923662ec:::rbd_data.115ee4742374.000000000000000a:head v 49'88, mlcod=48'86)", 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.889846224999999, 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062734230000000002, 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.871 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973500+0000", 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973502+0000", 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.04e-06 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973677+0000", 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473200000000001 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973714+0000", 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7117999999999999e-05 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973734+0000", 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0102000000000002e-05 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973879+0000", 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145654 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.973946+0000", 2024-04-24T00:35:27.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6476999999999994e-05 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979687+0000", 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057411140000000003 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:22.979773+0000", 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6186000000000005e-05 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.280264969000001, 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:27.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:35:27.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:27.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.280191113999997, 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:27.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:35:27.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:27.886 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.583496330000003, 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:35:27.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.777964371000003, 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:35:27.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:35:27.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:68687 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'253, mlcod=94'249)", 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.197952927999999, 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005827128, 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665393+0000", 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665394+0000", 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7700000000000004e-07 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665402+0000", 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4030000000000007e-06 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665414+0000", 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1467999999999999e-05 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665419+0000", 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1780000000000002e-06 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665485+0000", 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5944000000000004e-05 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.665507+0000", 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2359999999999999e-05 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.671176+0000", 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056689130000000003 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.671220+0000", 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4184999999999998e-05 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:69871 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'404, mlcod=94'396)", 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.030875601, 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055042449999999996, 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832470+0000", 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832471+0000", 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9200000000000001e-07 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832475+0000", 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.873e-06 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832482+0000", 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:35:27.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832486+0000", 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2470000000000002e-06 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832537+0000", 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1233000000000002e-05 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.832563+0000", 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5269000000000002e-05 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.837951+0000", 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053882720000000004 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.837975+0000", 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3683e-05 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:69977 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'424, mlcod=94'419)", 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.011119557000001, 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056478099999999996, 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852226+0000", 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852227+0000", 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-04-24T00:35:27.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852231+0000", 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1489999999999996e-06 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852240+0000", 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0010000000000007e-06 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852245+0000", 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.498e-06 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852283+0000", 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8195000000000003e-05 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.852298+0000", 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4911e-05 2024-04-24T00:35:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.857827+0000", 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055296210000000002 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.857874+0000", 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6897999999999997e-05 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70740 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'523, mlcod=94'518)", 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.897965955, 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061709410000000001, 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:27.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965380+0000", 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965381+0000", 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.094e-06 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965389+0000", 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9580000000000007e-06 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965423+0000", 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3787e-05 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965443+0000", 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0390999999999998e-05 2024-04-24T00:35:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965583+0000", 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013995500000000001 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.965606+0000", 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3003999999999999e-05 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971482+0000", 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058757649999999998 2024-04-24T00:35:27.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971551+0000", 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8986999999999995e-05 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70743 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'524, mlcod=94'518)", 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.897261627000001, 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055099820000000001, 2024-04-24T00:35:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966084+0000", 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966086+0000", 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1969999999999999e-06 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966096+0000", 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.995e-06 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966113+0000", 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7538e-05 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966124+0000", 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966181+0000", 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6937000000000001e-05 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.966199+0000", 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8002000000000001e-05 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971556+0000", 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053569029999999997 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.971594+0000", 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8186999999999999e-05 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70750 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'527, mlcod=94'519)", 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.894985055999999, 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005844307, 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968361+0000", 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968362+0000", 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4900000000000004e-07 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968371+0000", 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.619e-06 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968388+0000", 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7113999999999999e-05 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968401+0000", 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3495e-05 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968434+0000", 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3306000000000003e-05 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.968446+0000", 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1668000000000001e-05 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.974121+0000", 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056745290000000002 2024-04-24T00:35:27.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.974205+0000", 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4627000000000001e-05 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70804 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'539, mlcod=94'532)", 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.882686830000001, 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058417199999999999, 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980659+0000", 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980660+0000", 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.99e-07 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980665+0000", 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.95e-06 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980681+0000", 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6653000000000001e-05 2024-04-24T00:35:27.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980686+0000", 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9420000000000003e-06 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980873+0000", 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018610100000000001 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.980889+0000", 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6884e-05 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986449+0000", 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055591700000000004 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986501+0000", 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2321000000000003e-05 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70823 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'542, mlcod=94'536)", 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.880056736, 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057985090000000003, 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983289+0000", 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983290+0000", 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983296+0000", 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7969999999999997e-06 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983304+0000", 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4519999999999997e-06 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983309+0000", 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6809999999999997e-06 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983353+0000", 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3844999999999998e-05 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.983365+0000", 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.235e-05 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.988997+0000", 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005632176 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.989088+0000", 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0587999999999994e-05 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70843 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'546, mlcod=94'537)", 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.877319904, 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059306039999999999, 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986026+0000", 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986027+0000", 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2e-07 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986031+0000", 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986039+0000", 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1880000000000002e-06 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986043+0000", 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2339999999999996e-06 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986133+0000", 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9783000000000004e-05 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.986146+0000", 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2816000000000001e-05 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.991875+0000", 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057291429999999999 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:13.991957+0000", 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2027999999999995e-05 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70942 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'565, mlcod=94'560)", 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.858775176, 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005499884, 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004571+0000", 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004572+0000", 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.576e-06 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004583+0000", 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0383e-05 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004603+0000", 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9738999999999999e-05 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004612+0000", 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6919999999999997e-06 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004691+0000", 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8771999999999997e-05 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.004711+0000", 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0018e-05 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009995+0000", 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052835520000000004 2024-04-24T00:35:27.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.010071+0000", 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6151999999999999e-05 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70944 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'567, mlcod=94'561)", 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.856420635999999, 2024-04-24T00:35:27.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059596270000000003, 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006925+0000", 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006927+0000", 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006936+0000", 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6719999999999999e-06 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006960+0000", 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4249000000000001e-05 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.006975+0000", 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4259e-05 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.007069+0000", 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4477999999999999e-05 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.007088+0000", 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8332999999999999e-05 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012838+0000", 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057502830000000001 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012885+0000", 2024-04-24T00:35:27.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7206999999999997e-05 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70966 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'570, mlcod=94'563)", 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.854098383, 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059298320000000003, 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009248+0000", 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009249+0000", 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.155e-06 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009257+0000", 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.892e-06 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009275+0000", 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8272000000000001e-05 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009288+0000", 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3107e-05 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009373+0000", 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4561999999999994e-05 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.009407+0000", 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4100000000000002e-05 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.015084+0000", 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056774750000000004 2024-04-24T00:35:27.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.015177+0000", 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3269000000000004e-05 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:70979 3.1f e94/59 3:fb51b59a:::rbd_object_map.115eb6ef70cd.0000000000000024:head v 94'574, mlcod=94'565)", 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.851099078000001, 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054674650000000003, 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012247+0000", 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012248+0000", 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0920000000000001e-06 2024-04-24T00:35:27.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012256+0000", 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9119999999999998e-06 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012273+0000", 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7280000000000001e-05 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012295+0000", 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1866999999999999e-05 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012370+0000", 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5278999999999995e-05 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.012409+0000", 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8226999999999998e-05 2024-04-24T00:35:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.017649+0000", 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052406350000000004 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:14.017714+0000", 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5172999999999994e-05 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:77813 3.0 e102/21 3:05afdb1e:::rbd_data.115e805f80b3.0000000000000004:head v 102'237, mlcod=102'235)", 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.198524022, 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074761180000000003, 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664825+0000", 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664911+0000", 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5873999999999998e-05 2024-04-24T00:35:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664944+0000", 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664959+0000", 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5979000000000001e-05 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:27.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665112+0000", 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152943 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665157+0000", 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4725000000000003e-05 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672178+0000", 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070206390000000004 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:27.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672298+0000", 2024-04-24T00:35:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120308 2024-04-24T00:35:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:27.960 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:27.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:27.962 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:27.957+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:28.062 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:28.057+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:28.133 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:28.134 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:28.134 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:28.134 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:28.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:28.194 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:28.189+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:28.264 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:28.257+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:28.319 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:28.319 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:28.319 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:28.319 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:28.319 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:28.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:28.364 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:28.359+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:28.465 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:28.457+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970678950999996, 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970651777, 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970625671999997, 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:35:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:35:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970214690999995, 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:35:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:35:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970194882000001, 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:35:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:35:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970141823000006, 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970106967000007, 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:35:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:35:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970085402999999, 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:35:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:35:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970055911000003, 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:35:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970035049000003, 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:35:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:35:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.970012475000004, 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:35:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.287752976999997, 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:35:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.287402061999998, 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:35:28.569 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:28.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:28.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:35:28.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.280513741999997, 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:35:28.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:35:28.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44656 3.7 e79/44 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:head v 79'103, mlcod=77'101)", 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:28.580 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.605187275999999, 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026507441999999999, 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894327+0000", 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1e-07 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898194+0000", 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038672889999999999 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898203+0000", 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6340000000000007e-06 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898208+0000", 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2009999999999998e-06 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898262+0000", 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4466999999999999e-05 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898280+0000", 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7492000000000001e-05 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920764+0000", 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022483974 2024-04-24T00:35:28.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920834+0000", 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0174999999999998e-05 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44661 3.17 e79/59 3:eff1c653:::rbd_data.115ec7ed39de.0000000000000005:head v 79'91, mlcod=77'89)", 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.603319986999999, 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024065629000000002, 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8299999999999996e-07 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897771+0000", 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015761690000000001 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897786+0000", 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5051999999999999e-05 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897794+0000", 2024-04-24T00:35:28.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4689999999999997e-06 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897869+0000", 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.47e-05 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897899+0000", 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9777e-05 2024-04-24T00:35:28.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920181+0000", 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022282560999999999 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920259+0000", 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8218000000000002e-05 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.022232116000001, 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:35:28.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:35:28.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:28.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:35:28.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.011955700000001, 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:35:28.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:35:28.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78058 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-24T00:35:28.602 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.692620+0000", 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.80689346399999995, 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039546825000000001, 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78058 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692620+0000", 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692620+0000", 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692622+0000", 2024-04-24T00:35:28.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9889999999999999e-06 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692633+0000", 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0579e-05 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692657+0000", 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4000000000000001e-05 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692667+0000", 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0404999999999999e-05 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692735+0000", 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8139000000000006e-05 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692866+0000", 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013099300000000001 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.695390+0000", 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002524016 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.731846+0000", 2024-04-24T00:35:28.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036455631000000002 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.732039+0000", 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019255300000000001 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.732110+0000", 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1088999999999996e-05 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.732167+0000", 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7431e-05 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:78058 3.8 e105/21 ondisk, result = 0)", 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.702066+0000", 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.797448088, 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029791069, 2024-04-24T00:35:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702066+0000", 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:28.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702066+0000", 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702067+0000", 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4169999999999999e-06 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702076+0000", 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7090000000000008e-06 2024-04-24T00:35:28.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702097+0000", 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1637e-05 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702109+0000", 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1875999999999999e-05 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:28.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.731826+0000", 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029716414 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.731838+0000", 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2218000000000001e-05 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.731857+0000", 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8797999999999999e-05 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:28.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:28.614 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:28.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:28.616 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:28.611+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:35:28.717 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:28.709+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:28.732 INFO:teuthology.orchestra.run.smithi064.stdout:2974 trunc from 0xdcd8ef1 to 0x81af50f 2024-04-24T00:35:28.754 INFO:teuthology.orchestra.run.smithi064.stdout:2975 read 0x69635b9 thru 0x696cf93 (0x99db bytes) 2024-04-24T00:35:28.754 INFO:teuthology.orchestra.run.smithi064.stdout:2977 write 0x6137fa9 thru 0x613f3f4 (0x744c bytes) 2024-04-24T00:35:28.758 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:28.758 INFO:teuthology.orchestra.run.smithi064.stdout:2979 write 0x2f196b4 thru 0x2f1b0c6 (0x1a13 bytes) 2024-04-24T00:35:28.764 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:28.764 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:28.764 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:28.764 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi064.stdout:2982 trunc from 0x81af50f to 0x28f9110 2024-04-24T00:35:28.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:28.818 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:28.813+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:28.902 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:28.902 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:28.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:28.918 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:28.915+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:29.020 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:29.013+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:29.042 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:29.042 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:29.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:29.042 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:29.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.042 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5227 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.397354819, 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035851842000000002, 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5227 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640738+0000", 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640740+0000", 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640749+0000", 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9250000000000001e-06 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640764+0000", 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5478e-05 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640777+0000", 2024-04-24T00:35:29.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2476e-05 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642110+0000", 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001333323 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676468+0000", 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034357981000000003 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676590+0000", 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012197900000000001 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.397278619999994, 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:35:29.047 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.396820446000007, 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:35:29.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.396763139000001, 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:35:29.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:35:29.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.396504172999997, 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:35:29.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:35:29.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.395582700999995, 2024-04-24T00:35:29.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:35:29.069 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.823767970999995, 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:29.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.075 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.808238721999999, 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.179515402000007, 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.080 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:29.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:29.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.164480275000003, 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:35:29.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.401982277000002, 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.090 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:29.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.388159319000003, 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.095 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:29.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.161018134000003, 2024-04-24T00:35:29.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:29.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.149228483999998, 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30704 3.1a 3:5e74e97a:::rbd_data.115e8210e51a.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:29.109 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.785736565000001, 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043351496000000003, 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30704 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252358+0000", 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1790000000000001e-06 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252364+0000", 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6549999999999998e-06 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252381+0000", 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6767000000000001e-05 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252390+0000", 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9959999999999999e-06 2024-04-24T00:35:29.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252492+0000", 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102563 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252668+0000", 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017553499999999999 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.254545+0000", 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001877037 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295422+0000", 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040876527000000003 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295513+0000", 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1838999999999999e-05 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295614+0000", 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010027900000000001 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295708+0000", 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4118999999999998e-05 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.155188420000002, 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.116 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33518 3.b e70/59 ondisk, result = 0)", 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.143864892000003, 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042187448000000002, 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894230+0000", 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7680000000000001e-06 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894239+0000", 2024-04-24T00:35:29.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3319999999999998e-06 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894266+0000", 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049000000000001e-05 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894281+0000", 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5245000000000001e-05 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936247+0000", 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041965833000000001 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936257+0000", 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936415+0000", 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158605 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33734 3.b 3:d4ebe596:::rbd_data.115ec4f9b3a7.000000000000000e:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.518822696999997, 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041668288999999997, 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33734 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519272+0000", 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9870000000000002e-06 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519282+0000", 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0203e-05 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519310+0000", 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7880000000000001e-05 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519324+0000", 2024-04-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519402+0000", 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7865000000000001e-05 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519481+0000", 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9103000000000005e-05 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.522670+0000", 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003189 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560812+0000", 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038141427999999998 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560831+0000", 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9820999999999999e-05 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560886+0000", 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5087999999999998e-05 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560938+0000", 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1993999999999997e-05 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.133 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44651 3.6 e79/59 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head v 79'72, mlcod=77'70)", 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.149115328999997, 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048315631999999997, 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892459+0000", 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034802259999999999 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892475+0000", 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5998000000000002e-05 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892485+0000", 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8679999999999994e-06 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892653+0000", 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016800800000000001 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892721+0000", 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8953000000000002e-05 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937243+0000", 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044521666000000001 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937293+0000", 2024-04-24T00:35:29.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0043000000000002e-05 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44659 3.1d e79/53 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head v 79'105, mlcod=77'103)", 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.141634146000001, 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038358476000000002, 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896460+0000", 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2399999999999997e-07 2024-04-24T00:35:29.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897721+0000", 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012617049999999999 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897744+0000", 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2982000000000001e-05 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897756+0000", 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1559999999999999e-05 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897836+0000", 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0434999999999998e-05 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897871+0000", 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5147000000000001e-05 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934696+0000", 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036824511999999997 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934817+0000", 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121311 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.142 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:29.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:29.145 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:29.137+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:29.246 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:29.243+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:29.271 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:29.272 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:29.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:29.378 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:29.375+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:29.448 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:29.441+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:29.462 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:29.462 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:29.462 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:29.462 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:29.462 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:29.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:29.549 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:29.541+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:29.604 INFO:teuthology.orchestra.run.smithi064.stdout:2983 write 0x7a77929 thru 0x7a78d71 (0x1449 bytes) 2024-04-24T00:35:29.626 INFO:teuthology.orchestra.run.smithi064.stdout:2984 writesame 0x8eb2502 thru 0x8ebf64b data_size 0x59(0xd14a bytes) 2024-04-24T00:35:29.626 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:29.626 INFO:teuthology.orchestra.run.smithi064.stdout:2987 write 0x4d57ca4 thru 0x4d58233 (0x590 bytes) 2024-04-24T00:35:29.628 INFO:teuthology.orchestra.run.smithi064.stdout:2988 write 0x8dd78b4 thru 0x8dda9cd (0x311a bytes) 2024-04-24T00:35:29.643 INFO:teuthology.orchestra.run.smithi064.stdout:2990 trunc from 0x8ebf64c to 0x6a473c2 2024-04-24T00:35:29.650 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.936772666000003, 2024-04-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:35:29.652 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.652 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.652 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:29.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.652 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:29.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.653 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:29.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:35:29.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:35:29.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:35:29.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:35:29.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:35:29.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:35:29.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:29.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:35:29.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:35:29.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:35:29.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:35:29.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:35:29.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:35:29.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:35:29.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:35:29.660 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.660 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.936674740999997, 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.662 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.662 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:35:29.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.662 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:35:29.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:29.668 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.577295208999999, 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:35:29.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30133 3.1f e66/59 ondisk, result = 0)", 2024-04-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.564701661999997, 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060466079999999998, 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080440+0000", 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080446+0000", 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6699999999999999e-06 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080465+0000", 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080476+0000", 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0937999999999999e-05 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140869+0000", 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060392759999999997 2024-04-24T00:35:29.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140882+0000", 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3131e-05 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140905+0000", 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2642000000000001e-05 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.678 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30705 3.2 3:45c09e0c:::rbd_data.115e8210e51a.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.392688747999998, 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063528001000000001, 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30705 2024-04-24T00:35:29.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252455+0000", 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.303e-06 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252477+0000", 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1970000000000001e-05 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252515+0000", 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7616e-05 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252536+0000", 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376999999999999e-05 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252696+0000", 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160192 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252945+0000", 2024-04-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024906400000000001 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.256753+0000", 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038072129999999998 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315658+0000", 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058905258000000002 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315790+0000", 2024-04-24T00:35:29.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013223999999999999 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315894+0000", 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103886 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315980+0000", 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5882000000000002e-05 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30705 3.2 e66/53 ondisk, result = 0)", 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.377476645000002, 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048330598000000002, 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267667+0000", 2024-04-24T00:35:29.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.96e-06 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267683+0000", 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6336000000000001e-05 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267730+0000", 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7079000000000001e-05 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267757+0000", 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6364e-05 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315724+0000", 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047967018 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315742+0000", 2024-04-24T00:35:29.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7935999999999999e-05 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315995+0000", 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025290499999999999 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.689 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33506 3.c 3:3787ff34:::rbd_data.115ec4f9b3a7.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.847092603999997, 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061037313000000003, 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33506 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.690 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798052+0000", 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:35:29.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798067+0000", 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5160000000000001e-05 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798107+0000", 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9946e-05 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798130+0000", 2024-04-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2637e-05 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798238+0000", 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010841299999999999 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798344+0000", 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010631199999999999 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:29.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.801407+0000", 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030623600000000001 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858616+0000", 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057209342000000003 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858879+0000", 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000262952 2024-04-24T00:35:29.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858992+0000", 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113429 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859085+0000", 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2925999999999998e-05 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33506 3.c e70/56 ondisk, result = 0)", 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.837459836999997, 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051419185999999999, 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807682+0000", 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807687+0000", 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-04-24T00:35:29.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807704+0000", 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7424000000000001e-05 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807714+0000", 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3400000000000004e-06 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858714+0000", 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051000248999999998 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858732+0000", 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7995999999999998e-05 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859100+0000", 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036787400000000002 2024-04-24T00:35:29.699 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35694 3.1b 3:dcf0eb61:::rbd_data.115ec5833ea9.00000000000000aa:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.725643986000001, 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049399446, 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35694 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919498+0000", 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919504+0000", 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919520+0000", 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5908000000000001e-05 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919532+0000", 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2964e-05 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919630+0000", 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7096999999999995e-05 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919734+0000", 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104128 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.922056+0000", 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023219149999999999 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968775+0000", 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046719274999999998 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968817+0000", 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-05 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968858+0000", 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1440999999999998e-05 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968896+0000", 2024-04-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7741000000000003e-05 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.754311221999998, 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:35:29.706 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:29.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44664 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 524288~131072 [fadvise_sequential+fadvise_nocache],write 1179648~1048576 [fadvise_sequential+fadvise_nocache],write 10616832~2048 [fadvise_sequential+fadvise_nocache],write 10627072~4096 [fadvise_sequential+fadvise_nocache],write 10639360~4096 [fadvise_sequential+fadvise_nocache],write 10651648~4096 [fadvise_sequential+fadvise_nocache],write 10663936~4096 [fadvise_sequential+fadvise_nocache],write 10676224~4096 [fadvise_sequential+fadvise_nocache],write 10688512~4096 [fadvise_sequential+fadvise_nocache],write 10700800~4096 [fadvise_sequential+fadvise_nocache],write 10713088~4096 [fadvise_sequential+fadvise_nocache],write 10725376~4096 [fadvise_sequential+fadvise_nocache],write 10737664~4096 [fadvise_sequential+fadvise_nocache],write 10754048~4096 [fadvise_sequential+fadvise_nocache],write 10766336~4096 [fadvise_sequential+fadvise_nocache],write 10778624~4096 [fadvise_sequential+fadvise_nocache],write 10790912~4096 [fadvise_sequential+fadvise_nocache],write 10803200~4096 [fadvise_sequential+fadvise_nocache],write 10815488~4096 [fadvise_sequential+fadvise_nocache],write 10827776~4096 [fadvise_sequential+fadvise_nocache],write 10840064~4096 [fadvise_sequential+fadvise_nocache],write 10852352~4096 [fadvise_sequential+fadvise_nocache],write 10864640~4096 [fadvise_sequential+fadvise_nocache],write 10876928~2048 [fadvise_sequential+fadvise_nocache],write 10881024~4096 [fadvise_sequential+fadvise_nocache],write 10893312~4096 [fadvise_sequential+fadvise_nocache],write 10905600~4096 [fadvise_sequential+fadvise_nocache],write 10917888~4096 [fadvise_sequential+fadvise_nocache],write 10930176~4096 [fadvise_sequential+fadvise_nocache],write 10942464~4096 [fadvise_sequential+fadvise_nocache],write 10954752~4096 [fadvise_sequential+fadvise_nocache],write 10967040~4096 [fadvise_sequential+fadvise_nocache],write 10979328~4096 [fadvise_sequential+fadvise_nocache],write 10991616~4096 [fadvise_sequential+fadvise_nocache],write 11003904~4096 [fadvise_sequential+fadvise_nocache],write 11010048~2048 [fadvise_sequential+fadvise_nocache],write 11020288~4096 [fadvise_sequential+fadvise_nocache],write 11032576~4096 [fadvise_sequential+fadvise_nocache],write 11044864~4096 [fadvise_sequential+fadvise_nocache],write 11057152~4096 [fadvise_sequential+fadvise_nocache],write 11069440~4096 [fadvise_sequential+fadvise_nocache],write 11081728~4096 [fadvise_sequential+fadvise_nocache],write 11094016~4096 [fadvise_sequential+fadvise_nocache],write 11106304~4096 [fadvise_sequential+fadvise_nocache],write 11118592~4096 [fadvise_sequential+fadvise_nocache],write 11130880~4096 [fadvise_sequential+fadvise_nocache],write 14417920~4096 [fadvise_sequential+fadvise_nocache],write 14426112~8192 [fadvise_sequential+fadvise_nocache],write 14438400~8192 [fadvise_sequential+fadvise_nocache],write 14450688~8192 [fadvise_sequential+fadvise_nocache],write 14462976~8192 [fadvise_sequential+fadvise_nocache],write 14475264~8192 [fadvise_sequential+fadvise_nocache],write 14487552~8192 [fadvise_sequential+fadvise_nocache],write 14499840~8192 [fadvise_sequential+fadvise_nocache],write 14512128~8192 [fadvise_sequential+fadvise_nocache],write 14524416~8192 [fadvise_sequential+fadvise_nocache],write 14536704~8192 [fadvise_sequential+fadvise_nocache],write 14553088~8192 [fadvise_sequential+fadvise_nocache],write 14565376~8192 [fadvise_sequential+fadvise_nocache],write 14577664~8192 [fadvise_sequential+fadvise_nocache],write 14589952~8192 [fadvise_sequential+fadvise_nocache],write 14602240~8192 [fadvise_sequential+fadvise_nocache],write 14614528~8192 [fadvise_sequential+fadvise_nocache],write 14626816~8192 [fadvise_sequential+fadvise_nocache],write 14639104~8192 [fadvise_sequential+fadvise_nocache],write 14651392~8192 [fadvise_sequential+fadvise_nocache],write 14663680~8192 [fadvise_sequential+fadvise_nocache],write 14675968~4096 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.742237901000003, 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768734, 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44664 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:29.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902904+0000", 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5179999999999999e-06 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.904804+0000", 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018996390000000001 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905393+0000", 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058872600000000003 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905408+0000", 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5073e-05 2024-04-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905482+0000", 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4284999999999998e-05 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905785+0000", 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030266100000000002 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.907249+0000", 2024-04-24T00:35:29.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001464899 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.932396+0000", 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025146379 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958308+0000", 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025912418999999999 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958527+0000", 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218297 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958671+0000", 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144838 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.718 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.719 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46463 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:1d [read 360448~8192 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:29.719 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:29.719 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.047061224, 2024-04-24T00:35:29.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042258337999999999, 2024-04-24T00:35:29.720 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.720 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.720 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.720 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.720 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.720 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.720 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46463 2024-04-24T00:35:29.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.721 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:29.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598079+0000", 2024-04-24T00:35:29.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598080+0000", 2024-04-24T00:35:29.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:35:29.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598083+0000", 2024-04-24T00:35:29.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6740000000000001e-06 2024-04-24T00:35:29.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598091+0000", 2024-04-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1089999999999998e-06 2024-04-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598097+0000", 2024-04-24T00:35:29.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5370000000000003e-06 2024-04-24T00:35:29.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.635329+0000", 2024-04-24T00:35:29.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037231775000000002 2024-04-24T00:35:29.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.635418+0000", 2024-04-24T00:35:29.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8629999999999997e-05 2024-04-24T00:35:29.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640338+0000", 2024-04-24T00:35:29.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049199730000000002 2024-04-24T00:35:29.728 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.728 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.729 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46464 3.16 3.4814dcb6 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:29.729 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:29.729 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.047035301000001, 2024-04-24T00:35:29.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042285080000000003, 2024-04-24T00:35:29.729 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.730 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.730 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:29.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.731 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46464 2024-04-24T00:35:29.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.731 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:29.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598105+0000", 2024-04-24T00:35:29.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598106+0000", 2024-04-24T00:35:29.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7700000000000003e-07 2024-04-24T00:35:29.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598109+0000", 2024-04-24T00:35:29.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8080000000000001e-06 2024-04-24T00:35:29.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598117+0000", 2024-04-24T00:35:29.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2430000000000005e-06 2024-04-24T00:35:29.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.598124+0000", 2024-04-24T00:35:29.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6000000000000003e-06 2024-04-24T00:35:29.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640379+0000", 2024-04-24T00:35:29.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042255297999999997 2024-04-24T00:35:29.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.640390+0000", 2024-04-24T00:35:29.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1253999999999999e-05 2024-04-24T00:35:29.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.738 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.739 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:29.739 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:29.739 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.166551083000002, 2024-04-24T00:35:29.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:35:29.739 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.740 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.740 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.740 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:35:29.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.741 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:29.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:29.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:35:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:35:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:35:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:35:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:35:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:35:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:35:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:35:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:35:29.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:35:29.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:35:29.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:35:29.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:29.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:35:29.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:35:29.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:35:29.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:35:29.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:35:29.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:35:29.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.752 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.752 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.752 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:35:29.752 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:29.752 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.141282477000001, 2024-04-24T00:35:29.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:29.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.757 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:67305 3.17 e89/59 ondisk, result = 0)", 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.790508401, 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049974565999999998, 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854633+0000", 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854637+0000", 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.946e-06 2024-04-24T00:35:29.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854649+0000", 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2507000000000001e-05 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854658+0000", 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6330000000000002e-06 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904468+0000", 2024-04-24T00:35:29.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049809535000000002 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904479+0000", 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-05 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904607+0000", 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012738300000000001 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 11.288078756999999, 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.762 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:35:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 11.255473643, 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:29.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:35:29.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:35:29.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.850898948999999, 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:29.772 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.777 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.793058718999999, 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:29.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:29.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:35:29.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:29.781 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:29.782 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:29.782 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:35:29.782 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:35:29.782 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T00:35:29.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T00:35:29.783 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:29.645+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:29.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:29.785 INFO:teuthology.orchestra.run.smithi064.stdout:2991 trunc from 0x6a473c2 to 0x3f0486b 2024-04-24T00:35:29.785 INFO:teuthology.orchestra.run.smithi064.stdout:2992 read 0x1dccce6 thru 0x1dcfc8e (0x2fa9 bytes) 2024-04-24T00:35:29.785 INFO:teuthology.orchestra.run.smithi064.stdout:2993 punch from 0x7820d6 to 0x784eb1, (0x2ddb bytes) 2024-04-24T00:35:29.785 INFO:teuthology.orchestra.run.smithi064.stdout:2994 write 0xb5bae65 thru 0xb5bead4 (0x3c70 bytes) 2024-04-24T00:35:29.786 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:29.786 INFO:teuthology.orchestra.run.smithi064.stdout:2996 trunc from 0xb5bead5 to 0xd07170e 2024-04-24T00:35:29.786 INFO:teuthology.orchestra.run.smithi064.stdout:2997 trunc from 0xd07170e to 0x3d4f648 2024-04-24T00:35:29.786 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:29.777+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:29.812 INFO:teuthology.orchestra.run.smithi064.stdout:2998 read 0x2b269fd thru 0x2b364c9 (0xfacd bytes) 2024-04-24T00:35:29.812 INFO:teuthology.orchestra.run.smithi064.stdout:2999 punch from 0x2e17383 to 0x2e1a122, (0x2d9f bytes) 2024-04-24T00:35:29.825 INFO:teuthology.orchestra.run.smithi064.stdout:3001 writesame 0x4f25333 thru 0x4f27a37 data_size 0x1(0x2705 bytes) 2024-04-24T00:35:29.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:29.827 INFO:teuthology.orchestra.run.smithi064.stdout:3003 punch from 0x26b7c3e to 0x26c3a69, (0xbe2b bytes) 2024-04-24T00:35:29.827 INFO:teuthology.orchestra.run.smithi064.stdout:3004 punch from 0x23e1ede to 0x23ee6fa, (0xc81c bytes) 2024-04-24T00:35:29.828 INFO:teuthology.orchestra.run.smithi064.stdout:3005 read 0xab38d9 thru 0xac3627 (0xfd4f bytes) 2024-04-24T00:35:29.828 INFO:teuthology.orchestra.run.smithi064.stdout:3006 punch from 0x4149783 to 0x4156355, (0xcbd2 bytes) 2024-04-24T00:35:29.828 INFO:teuthology.orchestra.run.smithi064.stdout:3007 punch from 0x58a0ff to 0x5942e0, (0xa1e1 bytes) 2024-04-24T00:35:29.840 INFO:teuthology.orchestra.run.smithi064.stdout:3008 trunc from 0x4f27a38 to 0xcd5728f 2024-04-24T00:35:29.852 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:29.852 INFO:teuthology.orchestra.run.smithi064.stdout:3010 write 0x6ec8224 thru 0x6ed7729 (0xf506 bytes) 2024-04-24T00:35:29.857 INFO:teuthology.orchestra.run.smithi064.stdout:3011 read 0xc77f781 thru 0xc78485a (0x50da bytes) 2024-04-24T00:35:29.857 INFO:teuthology.orchestra.run.smithi064.stdout:3013 read 0x533fef1 thru 0x534bfc9 (0xc0d9 bytes) 2024-04-24T00:35:29.858 INFO:teuthology.orchestra.run.smithi064.stdout:3014 read 0x6f7c01f thru 0x6f8704d (0xb02f bytes) 2024-04-24T00:35:29.871 INFO:teuthology.orchestra.run.smithi064.stdout:3015 write 0xebafefe thru 0xebb49e4 (0x4ae7 bytes) 2024-04-24T00:35:29.879 INFO:teuthology.orchestra.run.smithi064.stdout:3018 read 0xd5f8d9b thru 0xd5fd311 (0x4577 bytes) 2024-04-24T00:35:29.879 INFO:teuthology.orchestra.run.smithi064.stdout:3019 writesame 0x754f4c1 thru 0x755ea7d data_size 0x85(0xf5bd bytes) 2024-04-24T00:35:29.882 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:29.882 INFO:teuthology.orchestra.run.smithi064.stdout:3022 write 0x1d167da thru 0x1d1fe99 (0x96c0 bytes) 2024-04-24T00:35:29.884 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:29.877+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:29.887 INFO:teuthology.orchestra.run.smithi064.stdout:3023 punch from 0x98b6d2f to 0x98b7b89, (0xe5a bytes) 2024-04-24T00:35:29.887 INFO:teuthology.orchestra.run.smithi064.stdout:3024 write 0x1593055 thru 0x1597e50 (0x4dfc bytes) 2024-04-24T00:35:29.890 INFO:teuthology.orchestra.run.smithi064.stdout:3025 read 0xbd4c4b9 thru 0xbd53811 (0x7359 bytes) 2024-04-24T00:35:29.891 INFO:teuthology.orchestra.run.smithi064.stdout:3026 read 0x16f8611 thru 0x1705c6d (0xd65d bytes) 2024-04-24T00:35:29.891 INFO:teuthology.orchestra.run.smithi064.stdout:3027 write 0xe7f1306 thru 0xe7fe188 (0xce83 bytes) 2024-04-24T00:35:29.896 INFO:teuthology.orchestra.run.smithi064.stdout:3028 write 0x573935b thru 0x574342a (0xa0d0 bytes) 2024-04-24T00:35:29.900 INFO:teuthology.orchestra.run.smithi064.stdout:3029 punch from 0xc3a7b6 to 0xc41b79, (0x73c3 bytes) 2024-04-24T00:35:29.900 INFO:teuthology.orchestra.run.smithi064.stdout:3031 read 0xe3da180 thru 0xe3e852c (0xe3ad bytes) 2024-04-24T00:35:29.912 INFO:teuthology.orchestra.run.smithi064.stdout:3032 trunc from 0xebb49e5 to 0x7a6aaa1 2024-04-24T00:35:29.921 INFO:teuthology.orchestra.run.smithi064.stdout:3033 read 0x5af6b1c thru 0x5afe4f2 (0x79d7 bytes) 2024-04-24T00:35:29.921 INFO:teuthology.orchestra.run.smithi064.stdout:3034 read 0x55a2b6 thru 0x560665 (0x63b0 bytes) 2024-04-24T00:35:29.922 INFO:teuthology.orchestra.run.smithi064.stdout:3035 punch from 0x796bcbc to 0x79755ad, (0x98f1 bytes) 2024-04-24T00:35:29.922 INFO:teuthology.orchestra.run.smithi064.stdout:3037 punch from 0x1b7448d to 0x1b78202, (0x3d75 bytes) 2024-04-24T00:35:29.923 INFO:teuthology.orchestra.run.smithi064.stdout:3038 read 0x70dd147 thru 0x70deebc (0x1d76 bytes) 2024-04-24T00:35:29.923 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:29.923 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:29.923 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:29.924 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:29.931 INFO:teuthology.orchestra.run.smithi064.stdout:3039 trunc from 0x7a6aaa1 to 0x1bfb529 2024-04-24T00:35:29.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:29.971 INFO:teuthology.orchestra.run.smithi064.stdout:3040 write 0xb10174f thru 0xb106ca7 (0x5559 bytes) 2024-04-24T00:35:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:3041 read 0x6d183e2 thru 0x6d1e921 (0x6540 bytes) 2024-04-24T00:35:29.976 INFO:teuthology.orchestra.run.smithi064.stdout:3042 trunc from 0xb106ca8 to 0xa736204 2024-04-24T00:35:29.984 INFO:teuthology.orchestra.run.smithi064.stdout:3043 punch from 0x823f649 to 0x8244fe9, (0x59a0 bytes) 2024-04-24T00:35:29.984 INFO:teuthology.orchestra.run.smithi064.stdout:3044 write 0x2645658 thru 0x2648e54 (0x37fd bytes) 2024-04-24T00:35:29.985 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:29.979+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:29.986 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:29.986 INFO:teuthology.orchestra.run.smithi064.stdout:3046 writesame 0x6b87020 thru 0x6b953a7 data_size 0x18(0xe388 bytes) 2024-04-24T00:35:29.989 INFO:teuthology.orchestra.run.smithi064.stdout:3048 writesame 0x655654b thru 0x65635a0 data_size 0x9(0xd056 bytes) 2024-04-24T00:35:30.003 INFO:teuthology.orchestra.run.smithi064.stdout:3049 writesame 0xb9ed6f3 thru 0xb9faf74 data_size 0x6b(0xd882 bytes) 2024-04-24T00:35:30.003 INFO:teuthology.orchestra.run.smithi064.stdout:3050 write 0x26500f8 thru 0x265f0f3 (0xeffc bytes) 2024-04-24T00:35:30.008 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.008 INFO:teuthology.orchestra.run.smithi064.stdout:3053 punch from 0xb9ac224 to 0xb9b1daf, (0x5b8b bytes) 2024-04-24T00:35:30.008 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.008 INFO:teuthology.orchestra.run.smithi064.stdout:3055 read 0x60d311d thru 0x60d51ef (0x20d3 bytes) 2024-04-24T00:35:30.009 INFO:teuthology.orchestra.run.smithi064.stdout:3056 read 0x1485d3 thru 0x14b38f (0x2dbd bytes) 2024-04-24T00:35:30.010 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.010 INFO:teuthology.orchestra.run.smithi064.stdout:3058 read 0xb5a733 thru 0xb63eaf (0x977d bytes) 2024-04-24T00:35:30.010 INFO:teuthology.orchestra.run.smithi064.stdout:3060 punch from 0x21d6c68 to 0x21df574, (0x890c bytes) 2024-04-24T00:35:30.010 INFO:teuthology.orchestra.run.smithi064.stdout:3061 write 0x3b3c8f3 thru 0x3b45393 (0x8aa1 bytes) 2024-04-24T00:35:30.011 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:30.011 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:30.016 INFO:teuthology.orchestra.run.smithi064.stdout:3062 trunc from 0xb9faf75 to 0x8daab0e 2024-04-24T00:35:30.024 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.025 INFO:teuthology.orchestra.run.smithi064.stdout:3064 read 0x61d98d thru 0x620e87 (0x34fb bytes) 2024-04-24T00:35:30.025 INFO:teuthology.orchestra.run.smithi064.stdout:3067 punch from 0x6fbe43d to 0x6fcd7b4, (0xf377 bytes) 2024-04-24T00:35:30.028 INFO:teuthology.orchestra.run.smithi064.stdout:3068 trunc from 0x8daab0e to 0xacce8e2 2024-04-24T00:35:30.029 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:30.030 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:30.036 INFO:teuthology.orchestra.run.smithi064.stdout:3069 writesame 0x1c6a96f thru 0x1c6ce81 data_size 0x1(0x2513 bytes) 2024-04-24T00:35:30.042 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.042 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.042 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.048 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:30.055 INFO:teuthology.orchestra.run.smithi064.stdout:3075 writesame 0xca3cccc thru 0xca405f4 data_size 0x1(0x3929 bytes) 2024-04-24T00:35:30.055 INFO:teuthology.orchestra.run.smithi064.stdout:3076 write 0xa118a20 thru 0xa12449c (0xba7d bytes) 2024-04-24T00:35:30.062 INFO:teuthology.orchestra.run.smithi064.stdout:3077 write 0xa767a4f thru 0xa77773b (0xfced bytes) 2024-04-24T00:35:30.066 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:30.066 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:30.066 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:30.067 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:30.067 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:30.067 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:30.068 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:30.071 INFO:teuthology.orchestra.run.smithi064.stdout:3078 punch from 0x7e090e2 to 0x7e0e7db, (0x56f9 bytes) 2024-04-24T00:35:30.071 INFO:teuthology.orchestra.run.smithi064.stdout:3079 write 0x160cb9d thru 0x161328c (0x66f0 bytes) 2024-04-24T00:35:30.074 INFO:teuthology.orchestra.run.smithi064.stdout:3080 write 0x738b087 thru 0x73929f8 (0x7972 bytes) 2024-04-24T00:35:30.077 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.077 INFO:teuthology.orchestra.run.smithi064.stdout:3082 writesame 0x1b4130f thru 0x1b47ed8 data_size 0x2a(0x6bca bytes) 2024-04-24T00:35:30.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:30.095 INFO:teuthology.orchestra.run.smithi064.stdout:3084 trunc from 0xca405f5 to 0x32eccb3 2024-04-24T00:35:30.095 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:30.095 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:30.115 INFO:teuthology.orchestra.run.smithi064.stdout:3085 punch from 0x11c1180 to 0x11ceeae, (0xdd2e bytes) 2024-04-24T00:35:30.121 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:30.122 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:30.126 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:30.123+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:30.130 INFO:teuthology.orchestra.run.smithi064.stdout:3087 writesame 0x860874d thru 0x86116b1 data_size 0x1(0x8f65 bytes) 2024-04-24T00:35:30.130 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.130 INFO:teuthology.orchestra.run.smithi064.stdout:3093 writesame 0x72636f thru 0x733a09 data_size 0x1(0xd69b bytes) 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 108, 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-04-24T00:33:24.857711+0000", 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-04-24T00:35:29.546350+0000", 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-04-24T00:35:14.574643+0000", 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-04-24T00:35:24.656453+0000", 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5801984, 2024-04-24T00:35:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 5, 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 3, 2024-04-24T00:35:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:39.734814+0000", 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:35:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-04-24T00:35:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "17", 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:35:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:35:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:35:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:35:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 8, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 8, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.25, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:43.786311+0000", 2024-04-24T00:35:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:35:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "20", 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:35:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:35:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:35:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.25, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.25, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "pool_client.0", 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:51.288702+0000", 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-04-24T00:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "101", 2024-04-24T00:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 40, 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 101, 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.75, 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.75, 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.059999942779541, 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.059999942779541, 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:35:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1c80b3f4-99ee-4725-90a8-920116161bf5", 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 13, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 74, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 95, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 53, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 75, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:35:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6817", 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:35:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6818", 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6819", 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6822", 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6823", 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6820", 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6821", 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6817/218259947", 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6819/218259947", 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6823/218259947", 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6821/218259947", 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "20e1331a-f8a2-48fd-b2b8-b7a92f21d78e", 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:35:30.163 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 106, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.164 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6824", 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6825", 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6826", 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6827", 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6830", 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6831", 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:35:30.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.168 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:35:30.168 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.168 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6828", 2024-04-24T00:35:30.168 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6829", 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6825/637720411", 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6827/637720411", 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6831/637720411", 2024-04-24T00:35:30.169 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6829/637720411", 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a51557a0-9634-4a20-82b0-b54cdd36b166", 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:35:30.170 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 55, 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/820024667", 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6811/820024667", 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6815/820024667", 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6813/820024667", 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "aafffa79-f532-4441-b1d6-f6441c5eebf8", 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.3392791748046875, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 106, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:35:30.176 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.177 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.178 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:35:30.179 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.180 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/1974534323", 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6803/1974534323", 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1974534323", 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1974534323", 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "78f05556-598e-4be2-b682-e3f58863eb82", 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:35:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 107, 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6816", 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6818", 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6820", 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6821", 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6826", 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6828", 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6823", 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6824", 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6818/1642543604", 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6821/1642543604", 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6828/1642543604", 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6824/1642543604", 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:35:30.186 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5", 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 106, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:35:30.187 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6808", 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6809", 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6810", 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6811", 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6814", 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6815", 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6812", 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6813", 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6809/172726194", 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6811/172726194", 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6815/172726194", 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6813/172726194", 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:35:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "6dad12bf-94de-482a-a402-49e67d5e5dc7", 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 106, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.193 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6800", 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6801", 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6802", 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6803", 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.195 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6806", 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6807", 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6804", 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6805", 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6801/2195746987", 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6803/2195746987", 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2195746987", 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6805/2195746987", 2024-04-24T00:35:30.197 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e6734cfe-3c24-4ddd-8ef0-5ed450efe772", 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:35:30.198 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 62, 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6817", 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6819", 2024-04-24T00:35:30.199 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6822", 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6825", 2024-04-24T00:35:30.200 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6830", 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6831", 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:35:30.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6827", 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6829", 2024-04-24T00:35:30.202 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6819/954791674", 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6825/954791674", 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6831/954791674", 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6829/954791674", 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-04-24T00:35:30.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.509233+0000", 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 75 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:35:30.204 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.465696+0000", 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.405177+0000", 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.269202+0000", 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:35:30.206 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.359180+0000", 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:35:30.207 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.351834+0000", 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.490127+0000", 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.208 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.484842+0000", 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.4", 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:35:30.209 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.1b", 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-04-24T00:35:30.210 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-04-24T00:35:30.211 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [ 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 36, 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 38, 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 40, 2024-04-24T00:35:30.212 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:30.213 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:30.214 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:30.214 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:30.216 INFO:tasks.thrashosds.thrasher:j is {'epoch': 108, 'fsid': '6a70d87d-4453-47be-af2a-5590f9851254', 'created': '2024-04-24T00:33:24.857711+0000', 'modified': '2024-04-24T00:35:29.546350+0000', 'last_up_change': '2024-04-24T00:35:14.574643+0000', 'last_in_change': '2024-04-24T00:35:24.656453+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T00:33:39.734814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T00:33:43.786311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-24T00:33:51.288702+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 40, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 0.75, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '1c80b3f4-99ee-4725-90a8-920116161bf5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 74, 'up_from': 95, 'up_thru': 53, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 218259947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 218259947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 218259947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 218259947}]}, 'public_addr': '172.21.15.64:6817/218259947', 'cluster_addr': '172.21.15.64:6819/218259947', 'heartbeat_back_addr': '172.21.15.64:6823/218259947', 'heartbeat_front_addr': '172.21.15.64:6821/218259947', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 637720411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 637720411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 637720411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 637720411}]}, 'public_addr': '172.21.15.64:6825/637720411', 'cluster_addr': '172.21.15.64:6827/637720411', 'heartbeat_back_addr': '172.21.15.64:6831/637720411', 'heartbeat_front_addr': '172.21.15.64:6829/637720411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a51557a0-9634-4a20-82b0-b54cdd36b166', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 820024667}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 820024667}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 820024667}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 820024667}]}, 'public_addr': '172.21.15.64:6809/820024667', 'cluster_addr': '172.21.15.64:6811/820024667', 'heartbeat_back_addr': '172.21.15.64:6815/820024667', 'heartbeat_front_addr': '172.21.15.64:6813/820024667', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aafffa79-f532-4441-b1d6-f6441c5eebf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3392791748046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1974534323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1974534323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1974534323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1974534323}]}, 'public_addr': '172.21.15.64:6801/1974534323', 'cluster_addr': '172.21.15.64:6803/1974534323', 'heartbeat_back_addr': '172.21.15.64:6807/1974534323', 'heartbeat_front_addr': '172.21.15.64:6805/1974534323', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '78f05556-598e-4be2-b682-e3f58863eb82', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6816', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6818', 'nonce': 1642543604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6820', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6821', 'nonce': 1642543604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6826', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6828', 'nonce': 1642543604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6823', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6824', 'nonce': 1642543604}]}, 'public_addr': '172.21.15.167:6818/1642543604', 'cluster_addr': '172.21.15.167:6821/1642543604', 'heartbeat_back_addr': '172.21.15.167:6828/1642543604', 'heartbeat_front_addr': '172.21.15.167:6824/1642543604', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6808', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 172726194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6810', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 172726194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6814', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 172726194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6812', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 172726194}]}, 'public_addr': '172.21.15.167:6809/172726194', 'cluster_addr': '172.21.15.167:6811/172726194', 'heartbeat_back_addr': '172.21.15.167:6815/172726194', 'heartbeat_front_addr': '172.21.15.167:6813/172726194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6dad12bf-94de-482a-a402-49e67d5e5dc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6800', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2195746987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6802', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2195746987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6806', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2195746987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6804', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2195746987}]}, 'public_addr': '172.21.15.167:6801/2195746987', 'cluster_addr': '172.21.15.167:6803/2195746987', 'heartbeat_back_addr': '172.21.15.167:6807/2195746987', 'heartbeat_front_addr': '172.21.15.167:6805/2195746987', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e6734cfe-3c24-4ddd-8ef0-5ed450efe772', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6817', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6819', 'nonce': 954791674}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6822', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6825', 'nonce': 954791674}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6830', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6831', 'nonce': 954791674}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6827', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6829', 'nonce': 954791674}]}, 'public_addr': '172.21.15.167:6819/954791674', 'cluster_addr': '172.21.15.167:6825/954791674', 'heartbeat_back_addr': '172.21.15.167:6831/954791674', 'heartbeat_front_addr': '172.21.15.167:6829/954791674', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.509233+0000', 'dead_epoch': 75}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.465696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.405177+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.269202+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.359180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.351834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.490127+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.484842+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.4', 'osds': [5, 7]}, {'pgid': '3.1b', 'osds': [5, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 36, 'length': 1}, {'begin': 38, 'length': 1}, {'begin': 40, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T00:35:30.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:35:30.217 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:30.217 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:30.218 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:30.209+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:30.232 INFO:teuthology.orchestra.run.smithi064.stdout:3094 read 0x3083577 thru 0x308db0f (0xa599 bytes) 2024-04-24T00:35:30.234 INFO:teuthology.orchestra.run.smithi064.stdout:3095 read 0x6817e6a thru 0x6823824 (0xb9bb bytes) 2024-04-24T00:35:30.242 INFO:teuthology.orchestra.run.smithi064.stdout:3096 trunc from 0x86116b2 to 0xdd3bf7b 2024-04-24T00:35:30.249 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:30.250 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:30.250 INFO:teuthology.orchestra.run.smithi064.stdout:3097 punch from 0xd88c2bd to 0xd88d3a4, (0x10e7 bytes) 2024-04-24T00:35:30.250 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.251 INFO:teuthology.orchestra.run.smithi064.stdout:3099 write 0x79e03f5 thru 0x79e9ef2 (0x9afe bytes) 2024-04-24T00:35:30.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:30.276 INFO:teuthology.orchestra.run.smithi064.stdout:3101 trunc from 0xdd3bf7b to 0x11d4048 2024-04-24T00:35:30.300 INFO:teuthology.orchestra.run.smithi064.stdout:3102 read 0xec5b0c thru 0xec6840 (0xd35 bytes) 2024-04-24T00:35:30.316 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:30.311+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:30.316 INFO:teuthology.orchestra.run.smithi064.stdout:3103 writesame 0x5c1f093 thru 0x5c26608 data_size 0x61(0x7576 bytes) 2024-04-24T00:35:30.332 INFO:teuthology.orchestra.run.smithi064.stdout:3105 trunc from 0x5c26609 to 0xe90bd29 2024-04-24T00:35:30.341 INFO:teuthology.orchestra.run.smithi064.stdout:3106 read 0xb08940b thru 0xb092fa4 (0x9b9a bytes) 2024-04-24T00:35:30.342 INFO:teuthology.orchestra.run.smithi064.stdout:3107 writesame 0x1d24073 thru 0x1d31666 data_size 0x2(0xd5f4 bytes) 2024-04-24T00:35:30.353 INFO:teuthology.orchestra.run.smithi064.stdout:3108 read 0x160f3e4 thru 0x161e610 (0xf22d bytes) 2024-04-24T00:35:30.354 INFO:teuthology.orchestra.run.smithi064.stdout:3109 writesame 0x374d49d thru 0x3752d40 data_size 0x6(0x58a4 bytes) 2024-04-24T00:35:30.358 INFO:teuthology.orchestra.run.smithi064.stdout:3111 writesame 0x970ed7e thru 0x97155e2 data_size 0x5(0x6865 bytes) 2024-04-24T00:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout:3116 write 0x8fcfe38 thru 0x8fd267f (0x2848 bytes) 2024-04-24T00:35:30.370 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.370 INFO:teuthology.orchestra.run.smithi064.stdout:3118 write 0x57ab85b thru 0x57b1c09 (0x63af bytes) 2024-04-24T00:35:30.373 INFO:teuthology.orchestra.run.smithi064.stdout:3120 read 0xa3f30fe thru 0xa3f4f17 (0x1e1a bytes) 2024-04-24T00:35:30.390 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:30.390 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:30.390 INFO:teuthology.orchestra.run.smithi064.stdout:3122 trunc from 0xe90bd29 to 0x351492f 2024-04-24T00:35:30.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:30.408 INFO:teuthology.orchestra.run.smithi064.stdout:3123 read 0x10769c4 thru 0x107fd91 (0x93ce bytes) 2024-04-24T00:35:30.409 INFO:teuthology.orchestra.run.smithi064.stdout:3124 trunc from 0x351492f to 0x4176860 2024-04-24T00:35:30.420 INFO:teuthology.orchestra.run.smithi064.stdout:3125 trunc from 0x4176860 to 0x6f05fdc 2024-04-24T00:35:30.427 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.427 INFO:teuthology.orchestra.run.smithi064.stdout:3127 punch from 0x5e52967 to 0x5e60501, (0xdb9a bytes) 2024-04-24T00:35:30.442 INFO:teuthology.orchestra.run.smithi064.stdout:3128 writesame 0xc791639 thru 0xc79ced0 data_size 0x21(0xb898 bytes) 2024-04-24T00:35:30.442 INFO:teuthology.orchestra.run.smithi064.stdout:3129 write 0x7e3b021 thru 0x7e40cfc (0x5cdc bytes) 2024-04-24T00:35:30.446 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.446 INFO:teuthology.orchestra.run.smithi064.stdout:3131 write 0x2b2d364 thru 0x2b379c1 (0xa65e bytes) 2024-04-24T00:35:30.450 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:30.447+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:30.451 INFO:teuthology.orchestra.run.smithi064.stdout:3132 read 0xb3f3a46 thru 0xb400c19 (0xd1d4 bytes) 2024-04-24T00:35:30.451 INFO:teuthology.orchestra.run.smithi064.stdout:3133 write 0xb998785 thru 0xb9a84de (0xfd5a bytes) 2024-04-24T00:35:30.462 INFO:teuthology.orchestra.run.smithi064.stdout:3134 trunc from 0xc79ced1 to 0xdcc1c7e 2024-04-24T00:35:30.470 INFO:teuthology.orchestra.run.smithi064.stdout:3136 read 0x7b51a83 thru 0x7b5c625 (0xaba3 bytes) 2024-04-24T00:35:30.470 INFO:teuthology.orchestra.run.smithi064.stdout:3137 write 0x9097cf9 thru 0x90a68e3 (0xebeb bytes) 2024-04-24T00:35:30.479 INFO:teuthology.orchestra.run.smithi064.stdout:3141 writesame 0x124c176 thru 0x1251e60 data_size 0x9(0x5ceb bytes) 2024-04-24T00:35:30.482 INFO:teuthology.orchestra.run.smithi064.stdout:3142 read 0x6534bc7 thru 0x653cdad (0x81e7 bytes) 2024-04-24T00:35:30.482 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.482 INFO:teuthology.orchestra.run.smithi064.stdout:3144 write 0xb408a24 thru 0xb40c877 (0x3e54 bytes) 2024-04-24T00:35:30.485 INFO:teuthology.orchestra.run.smithi064.stdout:3145 read 0x5be7a54 thru 0x5bedac1 (0x606e bytes) 2024-04-24T00:35:30.485 INFO:teuthology.orchestra.run.smithi064.stdout:3146 write 0x9b0fdbe thru 0x9b14f2a (0x516d bytes) 2024-04-24T00:35:30.492 INFO:teuthology.orchestra.run.smithi064.stdout:3148 read 0x32bfde3 thru 0x32c89b0 (0x8bce bytes) 2024-04-24T00:35:30.492 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.493 INFO:teuthology.orchestra.run.smithi064.stdout:3151 writesame 0x1658136 thru 0x1662956 data_size 0x1(0xa821 bytes) 2024-04-24T00:35:30.517 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:30.511+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:30.574 INFO:teuthology.orchestra.run.smithi064.stdout:3152 writesame 0xd46f497 thru 0xd47c9a3 data_size 0x1(0xd50d bytes) 2024-04-24T00:35:30.580 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:30.581 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:35:30.585 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:35:30.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:30.602 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-04-24T00:35:29.674392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":758161270,"num_objects":506,"num_object_clones":0,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":78504,"num_read_kb":423977,"num_write":12712,"num_write_kb":354082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":40086102,"num_keys_recovered":164,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9554,"ondisk_log_size":9554,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":267532,"kb_used_data":29120,"kb_used_omap":9,"kb_used_meta":238326,"kb_avail":565963508,"statfs":{"total":579820584960,"available":579546632192,"internally_reserved":0,"allocated":29818880,"data_stored":146138346,"data_compressed":7768822,"data_compressed_allocated":16494592,"data_compressed_original":139582500,"omap_allocated":9567,"internal_metadata":244046497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17910195,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1102,"num_read_kb":278,"num_write":413,"num_write_kb":1694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":505,"num_bytes_recovered":24968300,"num_keys_recovered":64,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997112"},"pg_stats":[{"pgid":"3.1f","version":"105'640","reported_seq":3125,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.716340+0000","last_change":"2024-04-24T00:35:21.693984+0000","last_active":"2024-04-24T00:35:28.716340+0000","last_peered":"2024-04-24T00:35:28.716340+0000","last_clean":"2024-04-24T00:35:28.716340+0000","last_became_active":"2024-04-24T00:34:34.562029+0000","last_became_peered":"2024-04-24T00:34:34.562029+0000","last_unstale":"2024-04-24T00:35:28.716340+0000","last_undegraded":"2024-04-24T00:35:28.716340+0000","last_fullsized":"2024-04-24T00:35:28.716340+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:35:58.845918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091050400000000002,"stat_sum":{"num_bytes":39964592,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4361,"num_read_kb":20933,"num_write":1024,"num_write_kb":12655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":107847,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.1e","version":"105'406","reported_seq":1987,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.710702+0000","last_change":"2024-04-24T00:35:21.704999+0000","last_active":"2024-04-24T00:35:28.710702+0000","last_peered":"2024-04-24T00:35:28.710702+0000","last_clean":"2024-04-24T00:35:28.710702+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:35:28.710702+0000","last_undegraded":"2024-04-24T00:35:28.710702+0000","last_fullsized":"2024-04-24T00:35:28.710702+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:42:53.287657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021739260000000001,"stat_sum":{"num_bytes":21004221,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2958,"num_read_kb":7549,"num_write":486,"num_write_kb":10100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.1d","version":"105'198","reported_seq":3407,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.716206+0000","last_change":"2024-04-24T00:35:21.694940+0000","last_active":"2024-04-24T00:35:28.716206+0000","last_peered":"2024-04-24T00:35:28.716206+0000","last_clean":"2024-04-24T00:35:28.716206+0000","last_became_active":"2024-04-24T00:34:28.726196+0000","last_became_peered":"2024-04-24T00:34:28.726196+0000","last_unstale":"2024-04-24T00:35:28.716206+0000","last_undegraded":"2024-04-24T00:35:28.716206+0000","last_fullsized":"2024-04-24T00:35:28.716206+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:50:38.609759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00161136,"stat_sum":{"num_bytes":32555141,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4023,"num_read_kb":31469,"num_write":251,"num_write_kb":12062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":458933,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.1c","version":"107'370","reported_seq":3084,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.715171+0000","last_change":"2024-04-24T00:35:21.693645+0000","last_active":"2024-04-24T00:35:28.715171+0000","last_peered":"2024-04-24T00:35:28.715171+0000","last_clean":"2024-04-24T00:35:28.715171+0000","last_became_active":"2024-04-24T00:34:31.747052+0000","last_became_peered":"2024-04-24T00:34:31.747052+0000","last_unstale":"2024-04-24T00:35:28.715171+0000","last_undegraded":"2024-04-24T00:35:28.715171+0000","last_fullsized":"2024-04-24T00:35:28.715171+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:13:47.893987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070963899999999997,"stat_sum":{"num_bytes":20205787,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":3108,"num_read_kb":14369,"num_write":436,"num_write_kb":8004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":949407,"num_keys_recovered":27,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.1b","version":"101'223","reported_seq":2514,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.714993+0000","last_change":"2024-04-24T00:35:21.694399+0000","last_active":"2024-04-24T00:35:28.714993+0000","last_peered":"2024-04-24T00:35:28.714993+0000","last_clean":"2024-04-24T00:35:28.714993+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:35:28.714993+0000","last_undegraded":"2024-04-24T00:35:28.714993+0000","last_fullsized":"2024-04-24T00:35:28.714993+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:28:22.785512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012923570000000001,"stat_sum":{"num_bytes":26704680,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":2794,"num_read_kb":19537,"num_write":255,"num_write_kb":12953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.1a","version":"97'162","reported_seq":2513,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.712668+0000","last_change":"2024-04-24T00:35:21.648239+0000","last_active":"2024-04-24T00:35:28.712668+0000","last_peered":"2024-04-24T00:35:28.712668+0000","last_clean":"2024-04-24T00:35:28.712668+0000","last_became_active":"2024-04-24T00:33:53.027055+0000","last_became_peered":"2024-04-24T00:33:53.027055+0000","last_unstale":"2024-04-24T00:35:28.712668+0000","last_undegraded":"2024-04-24T00:35:28.712668+0000","last_fullsized":"2024-04-24T00:35:28.712668+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:15.417861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00170283,"stat_sum":{"num_bytes":9349759,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":2115,"num_read_kb":11726,"num_write":191,"num_write_kb":9444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.19","version":"99'220","reported_seq":2079,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.702749+0000","last_change":"2024-04-24T00:35:21.653324+0000","last_active":"2024-04-24T00:35:27.702749+0000","last_peered":"2024-04-24T00:35:27.702749+0000","last_clean":"2024-04-24T00:35:27.702749+0000","last_became_active":"2024-04-24T00:33:53.017357+0000","last_became_peered":"2024-04-24T00:33:53.017357+0000","last_unstale":"2024-04-24T00:35:27.702749+0000","last_undegraded":"2024-04-24T00:35:27.702749+0000","last_fullsized":"2024-04-24T00:35:27.702749+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:48.624602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096005899999999998,"stat_sum":{"num_bytes":22788004,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1856,"num_read_kb":12769,"num_write":264,"num_write_kb":7276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.18","version":"105'146","reported_seq":2071,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.702454+0000","last_change":"2024-04-24T00:35:21.652953+0000","last_active":"2024-04-24T00:35:27.702454+0000","last_peered":"2024-04-24T00:35:27.702454+0000","last_clean":"2024-04-24T00:35:27.702454+0000","last_became_active":"2024-04-24T00:34:34.549969+0000","last_became_peered":"2024-04-24T00:34:34.549969+0000","last_unstale":"2024-04-24T00:35:27.702454+0000","last_undegraded":"2024-04-24T00:35:27.702454+0000","last_fullsized":"2024-04-24T00:35:27.702454+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:36.681288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065473300000000001,"stat_sum":{"num_bytes":26101937,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1758,"num_read_kb":17408,"num_write":130,"num_write_kb":8947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":803724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.17","version":"95'154","reported_seq":2418,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.716288+0000","last_change":"2024-04-24T00:35:21.694091+0000","last_active":"2024-04-24T00:35:28.716288+0000","last_peered":"2024-04-24T00:35:28.716288+0000","last_clean":"2024-04-24T00:35:28.716288+0000","last_became_active":"2024-04-24T00:34:34.561802+0000","last_became_peered":"2024-04-24T00:34:34.561802+0000","last_unstale":"2024-04-24T00:35:28.716288+0000","last_undegraded":"2024-04-24T00:35:28.716288+0000","last_fullsized":"2024-04-24T00:35:28.716288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:26:33.726278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010080969999999999,"stat_sum":{"num_bytes":14945483,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1949,"num_read_kb":15825,"num_write":188,"num_write_kb":7351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":934056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.16","version":"100'456","reported_seq":4132,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.724746+0000","last_change":"2024-04-24T00:35:28.724746+0000","last_active":"2024-04-24T00:35:28.724746+0000","last_peered":"2024-04-24T00:35:28.724746+0000","last_clean":"2024-04-24T00:35:28.724746+0000","last_became_active":"2024-04-24T00:35:28.721300+0000","last_became_peered":"2024-04-24T00:35:28.721300+0000","last_unstale":"2024-04-24T00:35:28.724746+0000","last_undegraded":"2024-04-24T00:35:28.724746+0000","last_fullsized":"2024-04-24T00:35:28.724746+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:08:36.888599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071159200000000004,"stat_sum":{"num_bytes":32925991,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5540,"num_read_kb":22057,"num_write":696,"num_write_kb":12389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.15","version":"105'262","reported_seq":4394,"reported_epoch":106,"state":"peering","last_fresh":"2024-04-24T00:35:27.705256+0000","last_change":"2024-04-24T00:35:27.705256+0000","last_active":"2024-04-24T00:35:27.705256+0000","last_peered":"2024-04-24T00:35:27.682210+0000","last_clean":"2024-04-24T00:35:24.604420+0000","last_became_active":"2024-04-24T00:35:26.725809+0000","last_became_peered":"2024-04-24T00:35:26.725809+0000","last_unstale":"2024-04-24T00:35:27.705256+0000","last_undegraded":"2024-04-24T00:35:27.705256+0000","last_fullsized":"2024-04-24T00:35:27.705256+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061413600000000004,"stat_sum":{"num_bytes":33068765,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4129,"num_read_kb":29005,"num_write":360,"num_write_kb":11023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":2312722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.14","version":"95'166","reported_seq":2578,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.725105+0000","last_change":"2024-04-24T00:35:28.725105+0000","last_active":"2024-04-24T00:35:28.725105+0000","last_peered":"2024-04-24T00:35:28.725105+0000","last_clean":"2024-04-24T00:35:28.725105+0000","last_became_active":"2024-04-24T00:35:28.722992+0000","last_became_peered":"2024-04-24T00:35:28.722992+0000","last_unstale":"2024-04-24T00:35:28.725105+0000","last_undegraded":"2024-04-24T00:35:28.725105+0000","last_fullsized":"2024-04-24T00:35:28.725105+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:36:25.310096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017093919999999999,"stat_sum":{"num_bytes":19765972,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2066,"num_read_kb":20603,"num_write":182,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1929866,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":190,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.710835+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:35:28.710835+0000","last_peered":"2024-04-24T00:35:28.710835+0000","last_clean":"2024-04-24T00:35:28.710835+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:35:28.710835+0000","last_undegraded":"2024-04-24T00:35:28.710835+0000","last_fullsized":"2024-04-24T00:35:28.710835+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"105'159","reported_seq":3093,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.703164+0000","last_change":"2024-04-24T00:35:21.653797+0000","last_active":"2024-04-24T00:35:27.703164+0000","last_peered":"2024-04-24T00:35:27.703164+0000","last_clean":"2024-04-24T00:35:27.703164+0000","last_became_active":"2024-04-24T00:34:34.550739+0000","last_became_peered":"2024-04-24T00:34:34.550739+0000","last_unstale":"2024-04-24T00:35:27.703164+0000","last_undegraded":"2024-04-24T00:35:27.703164+0000","last_fullsized":"2024-04-24T00:35:27.703164+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:07.033572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074492000000000004,"stat_sum":{"num_bytes":35730851,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2606,"num_read_kb":20642,"num_write":236,"num_write_kb":13255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1269922,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.703288+0000","last_change":"2024-04-24T00:33:44.961774+0000","last_active":"2024-04-24T00:35:27.703288+0000","last_peered":"2024-04-24T00:35:27.703288+0000","last_clean":"2024-04-24T00:35:27.703288+0000","last_became_active":"2024-04-24T00:33:44.960721+0000","last_became_peered":"2024-04-24T00:33:44.960721+0000","last_unstale":"2024-04-24T00:35:27.703288+0000","last_undegraded":"2024-04-24T00:35:27.703288+0000","last_fullsized":"2024-04-24T00:35:27.703288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:54.125887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"98'203","reported_seq":3081,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.729215+0000","last_change":"2024-04-24T00:35:28.728054+0000","last_active":"2024-04-24T00:35:28.729215+0000","last_peered":"2024-04-24T00:35:28.729215+0000","last_clean":"2024-04-24T00:35:28.729215+0000","last_became_active":"2024-04-24T00:35:28.721196+0000","last_became_peered":"2024-04-24T00:35:28.721196+0000","last_unstale":"2024-04-24T00:35:28.729215+0000","last_undegraded":"2024-04-24T00:35:28.729215+0000","last_fullsized":"2024-04-24T00:35:28.729215+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:06:02.477214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025800649999999999,"stat_sum":{"num_bytes":43812937,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2713,"num_read_kb":21001,"num_write":252,"num_write_kb":12633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4076209,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.713459+0000","last_change":"2024-04-24T00:35:25.694192+0000","last_active":"2024-04-24T00:35:28.713459+0000","last_peered":"2024-04-24T00:35:28.713459+0000","last_clean":"2024-04-24T00:35:28.713459+0000","last_became_active":"2024-04-24T00:35:25.693547+0000","last_became_peered":"2024-04-24T00:35:25.693547+0000","last_unstale":"2024-04-24T00:35:28.713459+0000","last_undegraded":"2024-04-24T00:35:28.713459+0000","last_fullsized":"2024-04-24T00:35:28.713459+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:32:47.752692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"105'269","reported_seq":1215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.703285+0000","last_change":"2024-04-24T00:35:21.653065+0000","last_active":"2024-04-24T00:35:27.703285+0000","last_peered":"2024-04-24T00:35:27.703285+0000","last_clean":"2024-04-24T00:35:27.703285+0000","last_became_active":"2024-04-24T00:33:53.015167+0000","last_became_peered":"2024-04-24T00:33:53.015167+0000","last_unstale":"2024-04-24T00:35:27.703285+0000","last_undegraded":"2024-04-24T00:35:27.703285+0000","last_fullsized":"2024-04-24T00:35:27.703285+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:52.838651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073859800000000003,"stat_sum":{"num_bytes":15625388,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1191,"num_read_kb":5496,"num_write":337,"num_write_kb":7777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.702734+0000","last_change":"2024-04-24T00:35:25.694365+0000","last_active":"2024-04-24T00:35:27.702734+0000","last_peered":"2024-04-24T00:35:27.702734+0000","last_clean":"2024-04-24T00:35:27.702734+0000","last_became_active":"2024-04-24T00:35:25.693906+0000","last_became_peered":"2024-04-24T00:35:25.693906+0000","last_unstale":"2024-04-24T00:35:27.702734+0000","last_undegraded":"2024-04-24T00:35:27.702734+0000","last_fullsized":"2024-04-24T00:35:27.702734+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:53:00.055840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"98'229","reported_seq":1507,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.715894+0000","last_change":"2024-04-24T00:35:21.694726+0000","last_active":"2024-04-24T00:35:28.715894+0000","last_peered":"2024-04-24T00:35:28.715894+0000","last_clean":"2024-04-24T00:35:28.715894+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:35:28.715894+0000","last_undegraded":"2024-04-24T00:35:28.715894+0000","last_fullsized":"2024-04-24T00:35:28.715894+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:31.229919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014980060000000001,"stat_sum":{"num_bytes":31195836,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1434,"num_read_kb":5863,"num_write":233,"num_write_kb":9346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.3","version":"105'164","reported_seq":1455,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.713511+0000","last_change":"2024-04-24T00:35:21.648393+0000","last_active":"2024-04-24T00:35:28.713511+0000","last_peered":"2024-04-24T00:35:28.713511+0000","last_clean":"2024-04-24T00:35:28.713511+0000","last_became_active":"2024-04-24T00:34:34.549707+0000","last_became_peered":"2024-04-24T00:34:34.549707+0000","last_unstale":"2024-04-24T00:35:28.713511+0000","last_undegraded":"2024-04-24T00:35:28.713511+0000","last_fullsized":"2024-04-24T00:35:28.713511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:41:50.535007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016438469999999999,"stat_sum":{"num_bytes":3134459,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1080,"num_read_kb":3486,"num_write":159,"num_write_kb":5193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":116,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.714724+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:35:28.714724+0000","last_peered":"2024-04-24T00:35:28.714724+0000","last_clean":"2024-04-24T00:35:28.714724+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:35:28.714724+0000","last_undegraded":"2024-04-24T00:35:28.714724+0000","last_fullsized":"2024-04-24T00:35:28.714724+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"105'244","reported_seq":1199,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.703093+0000","last_change":"2024-04-24T00:35:21.653645+0000","last_active":"2024-04-24T00:35:27.703093+0000","last_peered":"2024-04-24T00:35:27.703093+0000","last_clean":"2024-04-24T00:35:27.703093+0000","last_became_active":"2024-04-24T00:33:53.012086+0000","last_became_peered":"2024-04-24T00:33:53.012086+0000","last_unstale":"2024-04-24T00:35:27.703093+0000","last_undegraded":"2024-04-24T00:35:27.703093+0000","last_fullsized":"2024-04-24T00:35:27.703093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:02:29.352379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001221977,"stat_sum":{"num_bytes":3719192,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1433,"num_read_kb":5250,"num_write":232,"num_write_kb":5784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":183,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.697699+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:35:27.697699+0000","last_peered":"2024-04-24T00:35:27.697699+0000","last_clean":"2024-04-24T00:35:27.697699+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:35:27.697699+0000","last_undegraded":"2024-04-24T00:35:27.697699+0000","last_fullsized":"2024-04-24T00:35:27.697699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"105'170","reported_seq":1262,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.712035+0000","last_change":"2024-04-24T00:35:21.646967+0000","last_active":"2024-04-24T00:35:28.712035+0000","last_peered":"2024-04-24T00:35:28.712035+0000","last_clean":"2024-04-24T00:35:28.712035+0000","last_became_active":"2024-04-24T00:34:34.552406+0000","last_became_peered":"2024-04-24T00:34:34.552406+0000","last_unstale":"2024-04-24T00:35:28.712035+0000","last_undegraded":"2024-04-24T00:35:28.712035+0000","last_fullsized":"2024-04-24T00:35:28.712035+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:53.320742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063493399999999998,"stat_sum":{"num_bytes":16885173,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":786,"num_read_kb":9473,"num_write":196,"num_write_kb":8763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2075913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.703255+0000","last_change":"2024-04-24T00:35:25.694495+0000","last_active":"2024-04-24T00:35:27.703255+0000","last_peered":"2024-04-24T00:35:27.703255+0000","last_clean":"2024-04-24T00:35:27.703255+0000","last_became_active":"2024-04-24T00:35:25.694062+0000","last_became_peered":"2024-04-24T00:35:25.694062+0000","last_unstale":"2024-04-24T00:35:27.703255+0000","last_undegraded":"2024-04-24T00:35:27.703255+0000","last_fullsized":"2024-04-24T00:35:27.703255+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:38:43.882360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"101'304","reported_seq":900,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.714750+0000","last_change":"2024-04-24T00:35:21.695109+0000","last_active":"2024-04-24T00:35:28.714750+0000","last_peered":"2024-04-24T00:35:28.714750+0000","last_clean":"2024-04-24T00:35:28.714750+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:35:28.714750+0000","last_undegraded":"2024-04-24T00:35:28.714750+0000","last_fullsized":"2024-04-24T00:35:28.714750+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:43:07.237818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001671989,"stat_sum":{"num_bytes":22828543,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1531,"num_read_kb":5014,"num_write":367,"num_write_kb":10279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":116,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.714878+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:35:28.714878+0000","last_peered":"2024-04-24T00:35:28.714878+0000","last_clean":"2024-04-24T00:35:28.714878+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:35:28.714878+0000","last_undegraded":"2024-04-24T00:35:28.714878+0000","last_fullsized":"2024-04-24T00:35:28.714878+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":269,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.719733+0000","last_change":"2024-04-24T00:35:28.719733+0000","last_active":"2024-04-24T00:35:28.719733+0000","last_peered":"2024-04-24T00:35:28.719733+0000","last_clean":"2024-04-24T00:35:28.719733+0000","last_became_active":"2024-04-24T00:35:28.717573+0000","last_became_peered":"2024-04-24T00:35:28.717573+0000","last_unstale":"2024-04-24T00:35:28.719733+0000","last_undegraded":"2024-04-24T00:35:28.719733+0000","last_fullsized":"2024-04-24T00:35:28.719733+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:06:01.770652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"105'541","reported_seq":1888,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-24T00:35:27.731992+0000","last_change":"2024-04-24T00:35:21.648101+0000","last_active":"2024-04-24T00:35:27.731992+0000","last_peered":"2024-04-24T00:35:27.731992+0000","last_clean":"2024-04-24T00:35:27.731992+0000","last_became_active":"2024-04-24T00:33:53.022355+0000","last_became_peered":"2024-04-24T00:33:53.022355+0000","last_unstale":"2024-04-24T00:35:27.731992+0000","last_undegraded":"2024-04-24T00:35:27.731992+0000","last_fullsized":"2024-04-24T00:35:27.731992+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:52:20.997930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066699200000000004,"stat_sum":{"num_bytes":17360545,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":3062,"num_read_kb":7452,"num_write":728,"num_write_kb":12280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.9","version":"105'348","reported_seq":2370,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.712353+0000","last_change":"2024-04-24T00:35:21.648036+0000","last_active":"2024-04-24T00:35:28.712353+0000","last_peered":"2024-04-24T00:35:28.712353+0000","last_clean":"2024-04-24T00:35:28.712353+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:35:28.712353+0000","last_undegraded":"2024-04-24T00:35:28.712353+0000","last_fullsized":"2024-04-24T00:35:28.712353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:20:11.277542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015327839999999999,"stat_sum":{"num_bytes":41212442,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3115,"num_read_kb":9189,"num_write":429,"num_write_kb":8436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.a","version":"105'495","reported_seq":1212,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.710767+0000","last_change":"2024-04-24T00:35:21.704719+0000","last_active":"2024-04-24T00:35:28.710767+0000","last_peered":"2024-04-24T00:35:28.710767+0000","last_clean":"2024-04-24T00:35:28.710767+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:35:28.710767+0000","last_undegraded":"2024-04-24T00:35:28.710767+0000","last_fullsized":"2024-04-24T00:35:28.710767+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:07.241275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013730599999999999,"stat_sum":{"num_bytes":16183753,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":2159,"num_read_kb":6875,"num_write":770,"num_write_kb":16502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.b","version":"95'134","reported_seq":1074,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.711970+0000","last_change":"2024-04-24T00:35:21.647058+0000","last_active":"2024-04-24T00:35:28.711970+0000","last_peered":"2024-04-24T00:35:28.711970+0000","last_clean":"2024-04-24T00:35:28.711970+0000","last_became_active":"2024-04-24T00:34:34.549802+0000","last_became_peered":"2024-04-24T00:34:34.549802+0000","last_unstale":"2024-04-24T00:35:28.711970+0000","last_undegraded":"2024-04-24T00:35:28.711970+0000","last_fullsized":"2024-04-24T00:35:28.711970+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:21:18.256871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062814299999999995,"stat_sum":{"num_bytes":11945415,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1194,"num_read_kb":10729,"num_write":135,"num_write_kb":6354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":722108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.c","version":"102'496","reported_seq":4914,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.715229+0000","last_change":"2024-04-24T00:35:21.694615+0000","last_active":"2024-04-24T00:35:28.715229+0000","last_peered":"2024-04-24T00:35:28.715229+0000","last_clean":"2024-04-24T00:35:28.715229+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:35:28.715229+0000","last_undegraded":"2024-04-24T00:35:28.715229+0000","last_fullsized":"2024-04-24T00:35:28.715229+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:41:57.725437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013627489999999999,"stat_sum":{"num_bytes":62470277,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5807,"num_read_kb":34484,"num_write":745,"num_write_kb":20128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.d","version":"95'117","reported_seq":692,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.731150+0000","last_change":"2024-04-24T00:35:28.729993+0000","last_active":"2024-04-24T00:35:28.731150+0000","last_peered":"2024-04-24T00:35:28.731150+0000","last_clean":"2024-04-24T00:35:28.731150+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:35:28.731150+0000","last_undegraded":"2024-04-24T00:35:28.731150+0000","last_fullsized":"2024-04-24T00:35:28.731150+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:50.174344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022861309999999998,"stat_sum":{"num_bytes":3762622,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":264,"num_read_kb":2463,"num_write":129,"num_write_kb":6675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.e","version":"105'215","reported_seq":996,"reported_epoch":107,"state":"peering","last_fresh":"2024-04-24T00:35:28.714552+0000","last_change":"2024-04-24T00:35:28.714552+0000","last_active":"2024-04-24T00:35:28.714552+0000","last_peered":"2024-04-24T00:35:27.859160+0000","last_clean":"2024-04-24T00:35:24.504173+0000","last_became_active":"2024-04-24T00:35:27.140965+0000","last_became_peered":"2024-04-24T00:35:27.140965+0000","last_unstale":"2024-04-24T00:35:28.714552+0000","last_undegraded":"2024-04-24T00:35:28.714552+0000","last_fullsized":"2024-04-24T00:35:28.714552+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078362099999999997,"stat_sum":{"num_bytes":14068849,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":741,"num_read_kb":6522,"num_write":312,"num_write_kb":10188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":2728103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.f","version":"105'238","reported_seq":2126,"reported_epoch":107,"state":"peering","last_fresh":"2024-04-24T00:35:28.715685+0000","last_change":"2024-04-24T00:35:28.715685+0000","last_active":"2024-04-24T00:35:28.715685+0000","last_peered":"2024-04-24T00:35:27.715301+0000","last_clean":"2024-04-24T00:35:23.727436+0000","last_became_active":"2024-04-24T00:35:27.133786+0000","last_became_peered":"2024-04-24T00:35:27.133786+0000","last_unstale":"2024-04-24T00:35:28.715685+0000","last_undegraded":"2024-04-24T00:35:28.715685+0000","last_fullsized":"2024-04-24T00:35:28.715685+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046102300000000002,"stat_sum":{"num_bytes":20413798,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1828,"num_read_kb":8016,"num_write":310,"num_write_kb":17221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":2144359,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.10","version":"95'307","reported_seq":1527,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.711274+0000","last_change":"2024-04-24T00:35:21.705450+0000","last_active":"2024-04-24T00:35:28.711274+0000","last_peered":"2024-04-24T00:35:28.711274+0000","last_clean":"2024-04-24T00:35:28.711274+0000","last_became_active":"2024-04-24T00:34:34.557992+0000","last_became_peered":"2024-04-24T00:34:34.557992+0000","last_unstale":"2024-04-24T00:35:28.711274+0000","last_undegraded":"2024-04-24T00:35:28.711274+0000","last_fullsized":"2024-04-24T00:35:28.711274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:39.832837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018201999999999999,"stat_sum":{"num_bytes":10463442,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2208,"num_read_kb":7434,"num_write":383,"num_write_kb":7267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":179074,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.11","version":"105'334","reported_seq":1587,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.726015+0000","last_change":"2024-04-24T00:35:28.724032+0000","last_active":"2024-04-24T00:35:28.726015+0000","last_peered":"2024-04-24T00:35:28.726015+0000","last_clean":"2024-04-24T00:35:28.726015+0000","last_became_active":"2024-04-24T00:35:28.722568+0000","last_became_peered":"2024-04-24T00:35:28.722568+0000","last_unstale":"2024-04-24T00:35:28.726015+0000","last_undegraded":"2024-04-24T00:35:28.726015+0000","last_fullsized":"2024-04-24T00:35:28.726015+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:48:40.169992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057360400000000002,"stat_sum":{"num_bytes":29553948,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1497,"num_read_kb":4639,"num_write":377,"num_write_kb":11734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":1618865,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.12","version":"105'939","reported_seq":2452,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-04-24T00:35:27.697045+0000","last_change":"2024-04-24T00:35:28.705082+0000","last_active":"2024-04-24T00:35:28.705082+0000","last_peered":"2024-04-24T00:35:27.697045+0000","last_clean":"2024-04-24T00:35:24.604527+0000","last_became_active":"2024-04-24T00:35:27.167469+0000","last_became_peered":"2024-04-24T00:35:27.167469+0000","last_unstale":"2024-04-24T00:35:27.697045+0000","last_undegraded":"2024-04-24T00:35:27.697045+0000","last_fullsized":"2024-04-24T00:35:26.694310+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069715,"stat_sum":{"num_bytes":35244737,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4822,"num_read_kb":11479,"num_write":1595,"num_write_kb":21019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":5659027,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]},{"pgid":"3.13","version":"98'128","reported_seq":2785,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T00:35:28.725313+0000","last_change":"2024-04-24T00:35:28.725313+0000","last_active":"2024-04-24T00:35:28.725313+0000","last_peered":"2024-04-24T00:35:28.725313+0000","last_clean":"2024-04-24T00:35:28.725313+0000","last_became_active":"2024-04-24T00:35:28.722941+0000","last_became_peered":"2024-04-24T00:35:28.722941+0000","last_unstale":"2024-04-24T00:35:28.725313+0000","last_undegraded":"2024-04-24T00:35:28.725313+0000","last_fullsized":"2024-04-24T00:35:28.725313+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:26:38.456665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030194029999999999,"stat_sum":{"num_bytes":22578363,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2260,"num_read_kb":15120,"num_write":165,"num_write_kb":12182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":962033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"24","length":"1"},{"start":"26","length":"1"},{"start":"28","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":757570902,"num_objects":504,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":78388,"num_read_kb":423877,"num_write":12553,"num_write_kb":351666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":741,"num_bytes_recovered":39495734,"num_keys_recovered":164,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27086848,"data_stored":173666534,"data_compressed":9472411,"data_compressed_allocated":20127744,"data_compressed_original":170065126,"omap_allocated":0,"internal_metadata":0},"log_size":9437,"ondisk_log_size":9437,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":2361472,"data_compressed":26218,"data_compressed_allocated":147456,"data_compressed_original":2326528,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574875,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38864,"kb_used_data":3524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332976,"statfs":{"total":96636764160,"available":96596967424,"internally_reserved":0,"allocated":3608576,"data_stored":15577984,"data_compressed":741257,"data_compressed_allocated":1679360,"data_compressed_original":14629209,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45356,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94326484,"statfs":{"total":96636764160,"available":96590319616,"internally_reserved":0,"allocated":5996544,"data_stored":30040568,"data_compressed":1741577,"data_compressed_allocated":3551232,"data_compressed_original":28822694,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49612,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94322228,"statfs":{"total":96636764160,"available":96585961472,"internally_reserved":0,"allocated":6090752,"data_stored":33036579,"data_compressed":1543571,"data_compressed_allocated":3538944,"data_compressed_original":31761743,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51196,"kb_used_data":6132,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94320644,"statfs":{"total":96636764160,"available":96584339456,"internally_reserved":0,"allocated":6279168,"data_stored":31799552,"data_compressed":1883757,"data_compressed_allocated":3784704,"data_compressed_original":30558558,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37980,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333860,"statfs":{"total":96636764160,"available":96597872640,"internally_reserved":0,"allocated":2699264,"data_stored":7672336,"data_compressed":546639,"data_compressed_allocated":974848,"data_compressed_original":6836224,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44524,"kb_used_data":5024,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94327316,"statfs":{"total":96636764160,"available":96591171584,"internally_reserved":0,"allocated":5144576,"data_stored":28011327,"data_compressed":1312021,"data_compressed_allocated":2965504,"data_compressed_original":26974072,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":408021893124,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":7369185,"data_compressed":431142,"data_compressed_allocated":892928,"data_compressed_original":7188480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3842048,"data_stored":26820086,"data_compressed":1305036,"data_compressed_allocated":2928640,"data_compressed_original":26384248,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1437696,"data_stored":7071463,"data_compressed":546639,"data_compressed_allocated":974848,"data_compressed_original":6836224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5017600,"data_stored":31198679,"data_compressed":1883757,"data_compressed_allocated":3784704,"data_compressed_original":30558558,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4829184,"data_stored":32435706,"data_compressed":1543571,"data_compressed_allocated":3538944,"data_compressed_original":31761743,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4739072,"data_stored":29434206,"data_compressed":1741480,"data_compressed_allocated":3547136,"data_compressed_original":28814502,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2306048,"data_stored":14386743,"data_compressed":734272,"data_compressed_allocated":1642496,"data_compressed_original":14039385,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:35:30.605 INFO:tasks.thrashosds.thrasher:Setting 3.16 to [3, 2] 2024-04-24T00:35:30.605 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.16', '3', '2'] 2024-04-24T00:35:30.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.16 3 2 2024-04-24T00:35:30.638 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:30.635+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:30.664 INFO:teuthology.orchestra.run.smithi064.stdout:3153 read 0x35d2f22 thru 0x35d743f (0x451e bytes) 2024-04-24T00:35:30.664 INFO:teuthology.orchestra.run.smithi064.stdout:3155 write 0x26d5bde thru 0x26e3d81 (0xe1a4 bytes) 2024-04-24T00:35:30.669 INFO:teuthology.orchestra.run.smithi064.stdout:3156 punch from 0xe7eec7 to 0xe89353, (0xa48c bytes) 2024-04-24T00:35:30.669 INFO:teuthology.orchestra.run.smithi064.stdout:3158 read 0x83c7efc thru 0x83d5759 (0xd85e bytes) 2024-04-24T00:35:30.670 INFO:teuthology.orchestra.run.smithi064.stdout:3159 read 0xae29940 thru 0xae2fba2 (0x6263 bytes) 2024-04-24T00:35:30.678 INFO:teuthology.orchestra.run.smithi064.stdout:3160 write 0xebb82da thru 0xebb8c19 (0x940 bytes) 2024-04-24T00:35:30.679 INFO:teuthology.orchestra.run.smithi064.stdout:3161 read 0x5ac305 thru 0x5bad0a (0xea06 bytes) 2024-04-24T00:35:30.694 INFO:teuthology.orchestra.run.smithi064.stdout:3162 trunc from 0xebb8c1a to 0x562d249 2024-04-24T00:35:30.719 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.719 INFO:teuthology.orchestra.run.smithi064.stdout:3165 read 0x1630400 thru 0x163ac7a (0xa87b bytes) 2024-04-24T00:35:30.719 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:30.713+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:30.720 INFO:teuthology.orchestra.run.smithi064.stdout:3166 read 0x355d8fa thru 0x35686f2 (0xadf9 bytes) 2024-04-24T00:35:30.720 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:30.720 INFO:teuthology.orchestra.run.smithi064.stdout:3168 read 0x297df31 thru 0x298b067 (0xd137 bytes) 2024-04-24T00:35:30.737 INFO:teuthology.orchestra.run.smithi064.stdout:3169 write 0xdcd3ee1 thru 0xdce12b1 (0xd3d1 bytes) 2024-04-24T00:35:30.745 INFO:teuthology.orchestra.run.smithi064.stdout:3170 write 0x7f4045f thru 0x7f47f41 (0x7ae3 bytes) 2024-04-24T00:35:30.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:35:30.751 INFO:teuthology.orchestra.run.smithi064.stdout:3171 read 0xbec73ab thru 0xbec7864 (0x4ba bytes) 2024-04-24T00:35:30.751 INFO:teuthology.orchestra.run.smithi064.stdout:3172 read 0xcf8c19c thru 0xcf8faa3 (0x3908 bytes) 2024-04-24T00:35:30.751 INFO:teuthology.orchestra.run.smithi064.stdout:3174 read 0x48ce489 thru 0x48d0ef1 (0x2a69 bytes) 2024-04-24T00:35:30.751 INFO:teuthology.orchestra.run.smithi064.stdout:3175 clone 20 order 22 su 16384 sc 11 2024-04-24T00:35:30.765 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:30.765 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:30.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:30.846 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:30.841+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:30.911 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:30.911 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:30.920 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:30.913+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:30.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:31.021 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:31.017+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:31.085 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:31.085 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:31.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:31.123 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:31.119+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:31.224 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:31.217+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:31.263 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:31.264 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:31.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:31.366 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:31.361+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:31.425 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:31.423+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:31.447 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:31.447 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:31.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:31.558 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:31.553+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:31.628 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:31.621+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:31.636 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:31.636 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:31.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:31.716 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:35:31.720 INFO:teuthology.orchestra.run.smithi064.stderr:set 3.16 pg_upmap mapping to [3,2] 2024-04-24T00:35:31.729 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:31.721+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:31.821 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:31.821 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:31.830 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:31.825+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:31.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:31.930 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:31.927+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:32.006 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:32.006 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:32.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:32.066 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:32.061+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:32.132 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:32.125+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:32.144 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:32.145 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:32.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:32.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:32.234 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:32.227+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:32.324 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:32.324 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:32.334 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:32.329+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:32.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:32.389 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:32.390 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:32.407 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:32.407 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:32.424 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:32.424 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:32.435 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:32.429+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:32.441 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:32.441 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:32.462 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:32.462 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:32.483 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:32.483 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:32.502 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:32.502 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:32.502 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:32.502 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:32.505 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:32.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:32.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:32.527 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:32.528 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:32.547 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:32.541+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:32.637 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:32.629+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:32.646 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:32.647 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:32.647 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:32.647 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:32.647 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:32.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:32.738 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:32.735+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:32.839 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:32.835+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:32.840 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:32.840 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:32.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:32.840 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:32.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.840 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2058 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2752512~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306826384999994, 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023213880999999999, 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2058 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.841 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528835+0000", 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7399999999999999e-07 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528842+0000", 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3499999999999999e-06 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528850+0000", 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0400000000000004e-06 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528855+0000", 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.169e-06 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551874+0000", 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023018818999999999 2024-04-24T00:35:32.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.551950+0000", 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6890000000000004e-05 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552049+0000", 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8239000000000006e-05 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2059 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2883584~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306799210999998, 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023430294000000001, 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.847 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2059 2024-04-24T00:35:32.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.847 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:32.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528862+0000", 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4199999999999996e-07 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528869+0000", 2024-04-24T00:35:32.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5939999999999999e-06 2024-04-24T00:35:32.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528876+0000", 2024-04-24T00:35:32.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.759e-06 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528881+0000", 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1989999999999997e-06 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552094+0000", 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023212615999999998 2024-04-24T00:35:32.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552174+0000", 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0819e-05 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552292+0000", 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011776499999999999 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2060 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3014656~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306773105999994, 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023734424, 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2060 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528888+0000", 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528894+0000", 2024-04-24T00:35:32.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8209999999999998e-06 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528901+0000", 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8990000000000004e-06 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.528907+0000", 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4129999999999997e-06 2024-04-24T00:35:32.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552407+0000", 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023500437999999998 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552480+0000", 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3040000000000005e-05 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.552622+0000", 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142414 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2085 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2162688~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306362125000007, 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024438096999999999, 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2085 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529299+0000", 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6600000000000002e-07 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529301+0000", 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.945e-06 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529308+0000", 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529313+0000", 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9840000000000001e-06 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553488+0000", 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024175148 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553603+0000", 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115121 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553737+0000", 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013433499999999999 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2086 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2293760~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306342315999999, 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024730946, 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2086 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:32.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529319+0000", 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529321+0000", 2024-04-24T00:35:32.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8980000000000001e-06 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529327+0000", 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0959999999999997e-06 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529332+0000", 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8570000000000003e-06 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553782+0000", 2024-04-24T00:35:32.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024449453999999999 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.553883+0000", 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100877 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554050+0000", 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167238 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2087 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2424832~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306289257000003, 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025416500000000002, 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2087 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.866 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529372+0000", 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-04-24T00:35:32.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529375+0000", 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8320000000000002e-06 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529383+0000", 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4950000000000002e-06 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529396+0000", 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3444e-05 2024-04-24T00:35:32.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554556+0000", 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025159428000000001 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554663+0000", 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107423 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554788+0000", 2024-04-24T00:35:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012527099999999999 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2088 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2555904~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306254401000004, 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025631442000000001, 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2088 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529407+0000", 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2799999999999996e-07 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529410+0000", 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5160000000000001e-06 2024-04-24T00:35:32.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529416+0000", 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6630000000000004e-06 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529421+0000", 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8840000000000002e-06 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554828+0000", 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025406353 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.554926+0000", 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8684999999999995e-05 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555038+0000", 2024-04-24T00:35:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011181299999999999 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2089 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2686976~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306232836999996, 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025919273, 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2089 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529428+0000", 2024-04-24T00:35:32.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529429+0000", 2024-04-24T00:35:32.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529437+0000", 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529444+0000", 2024-04-24T00:35:32.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.545e-06 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529449+0000", 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8720000000000001e-06 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555074+0000", 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025625372 2024-04-24T00:35:32.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555173+0000", 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8881000000000004e-05 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555348+0000", 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2090 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2818048~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306203345, 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026158260999999999, 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2090 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:32.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529458+0000", 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4600000000000005e-07 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529461+0000", 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3199999999999998e-06 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529467+0000", 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4710000000000004e-06 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529472+0000", 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555387+0000", 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025915516 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555491+0000", 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103861 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555616+0000", 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012477300000000001 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2091 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 2949120~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306182483000001, 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026410883, 2024-04-24T00:35:32.886 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2091 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:32.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529479+0000", 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6699999999999999e-07 2024-04-24T00:35:32.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529481+0000", 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529489+0000", 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9070000000000007e-06 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529494+0000", 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2340000000000002e-06 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555657+0000", 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026162984 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555760+0000", 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010231200000000001 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.555890+0000", 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012990899999999999 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:2092 3.5 3:a66c4c97:::rbd_data.115eb9c76af6.0000000000000003:5 [sparse-read 3080192~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.306159909000002, 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027099261999999999, 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2092 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.892 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529501+0000", 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529502+0000", 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7899999999999999e-07 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529507+0000", 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529514+0000", 2024-04-24T00:35:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6420000000000001e-06 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.529519+0000", 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1889999999999998e-06 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556375+0000", 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026855818 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556481+0000", 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010600800000000001 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:04.556600+0000", 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119394 2024-04-24T00:35:32.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.623900410999994, 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:35:32.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:35:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.623549495999995, 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:35:32.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:32.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:35:32.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.910 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.616661175999994, 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:35:32.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44656 3.7 e79/44 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:head v 79'103, mlcod=77'101)", 2024-04-24T00:35:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.94133471, 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026507441999999999, 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894326+0000", 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.894327+0000", 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1e-07 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898194+0000", 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038672889999999999 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898203+0000", 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6340000000000007e-06 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898208+0000", 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2009999999999998e-06 2024-04-24T00:35:32.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898262+0000", 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4466999999999999e-05 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.898280+0000", 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7492000000000001e-05 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:32.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920764+0000", 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022483974 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920834+0000", 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0174999999999998e-05 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.920 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:44661 3.17 e79/59 3:eff1c653:::rbd_data.115ec7ed39de.0000000000000005:head v 79'91, mlcod=77'89)", 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.939467421000003, 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024065629000000002, 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:32.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.896194+0000", 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8299999999999996e-07 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897771+0000", 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015761690000000001 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897786+0000", 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5051999999999999e-05 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897794+0000", 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4689999999999997e-06 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897869+0000", 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.47e-05 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.897899+0000", 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9777e-05 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920181+0000", 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022282560999999999 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:56.920259+0000", 2024-04-24T00:35:32.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8218000000000002e-05 2024-04-24T00:35:32.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.926 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:32.926 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:32.926 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.358379549999999, 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:35:32.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.348103133999999, 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.933 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:35:32.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78058 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.692620+0000", 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1430408979999997, 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039546825000000001, 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78058 2024-04-24T00:35:32.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692620+0000", 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692620+0000", 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692622+0000", 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9889999999999999e-06 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692633+0000", 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0579e-05 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692657+0000", 2024-04-24T00:35:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4000000000000001e-05 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692667+0000", 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0404999999999999e-05 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692735+0000", 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8139000000000006e-05 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.692866+0000", 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013099300000000001 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.695390+0000", 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002524016 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.731846+0000", 2024-04-24T00:35:32.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036455631000000002 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.732039+0000", 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019255300000000001 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.732110+0000", 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1088999999999996e-05 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.732167+0000", 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7431e-05 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:78058 3.8 e105/21 ondisk, result = 0)", 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.702066+0000", 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1335955220000002, 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029791069, 2024-04-24T00:35:32.944 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702066+0000", 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702066+0000", 2024-04-24T00:35:32.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702067+0000", 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4169999999999999e-06 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702076+0000", 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7090000000000008e-06 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702097+0000", 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1637e-05 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.702109+0000", 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1875999999999999e-05 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:32.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.731826+0000", 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029716414 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.731838+0000", 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2218000000000001e-05 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.731857+0000", 2024-04-24T00:35:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8797999999999999e-05 2024-04-24T00:35:32.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:32.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:32.949 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:32.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:32.952 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:32.945+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:33.052 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:33.047+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:33.084 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:33.084 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:33.084 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:33.084 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:33.098 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:33.152 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:33.149+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:33.254 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:33.247+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:33.267 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:33.268 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:33.268 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:33.268 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:33.268 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:33.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:33.354 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:33.349+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:33.452 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:33.452 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:33.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:33.452 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:33.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.452 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5233 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:33.452 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.806628179000001, 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036133321000000003, 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5233 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640814+0000", 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640816+0000", 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3480000000000001e-06 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.640829+0000", 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662000000000001e-05 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641197+0000", 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036793900000000002 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641213+0000", 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5460000000000001e-05 2024-04-24T00:35:33.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676686+0000", 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035473574000000001 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676813+0000", 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012673200000000001 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676948+0000", 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013460600000000001 2024-04-24T00:35:33.456 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5237 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1048576~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.806170004999998, 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036211720000000003, 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.457 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5237 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641272+0000", 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641274+0000", 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641281+0000", 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4259999999999997e-06 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641298+0000", 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6636000000000001e-05 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641312+0000", 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4052e-05 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677011+0000", 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035698428999999997 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677158+0000", 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147851 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677484+0000", 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032574600000000002 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5241 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.806112698000007, 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036491184000000003, 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5241 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641330+0000", 2024-04-24T00:35:33.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641331+0000", 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641346+0000", 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4628e-05 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641354+0000", 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8519999999999993e-06 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641361+0000", 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7179999999999999e-06 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677549+0000", 2024-04-24T00:35:33.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036187812 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677643+0000", 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3678999999999995e-05 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677821+0000", 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017827300000000001 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5247 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.805853732000003, 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036746587999999997, 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5247 2024-04-24T00:35:33.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641589+0000", 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641590+0000", 2024-04-24T00:35:33.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.226e-06 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641595+0000", 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7879999999999997e-06 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641615+0000", 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0758e-05 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.641627+0000", 2024-04-24T00:35:33.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1464000000000001e-05 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.677927+0000", 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036300394999999999 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678142+0000", 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021435900000000001 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678335+0000", 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019359800000000001 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:5252 3.14 3:28ef9212:::rbd_data.115e5ff16b0c.0000000000000045:7 [sparse-read 1179648~0 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.804932260000001, 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036498916999999999, 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.470 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 5252 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642510+0000", 2024-04-24T00:35:33.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642512+0000", 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.642522+0000", 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7890000000000006e-06 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676453+0000", 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033930835999999999 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.676482+0000", 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8782999999999999e-05 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678427+0000", 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019450209999999999 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.678671+0000", 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000243945 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:08.679009+0000", 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033858 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.474 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.233117530000001, 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.217588281000005, 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.480 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.588864960999999, 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.484 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:33.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:33.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10591 3.1c e41/21 ondisk, result = 0)", 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.573829833999994, 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039163134000000002, 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.490 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873613+0000", 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873615+0000", 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9630000000000001e-06 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873624+0000", 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5089999999999999e-06 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873658+0000", 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4073999999999999e-05 2024-04-24T00:35:33.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.873676+0000", 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7762e-05 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912464+0000", 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038788017000000001 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912487+0000", 2024-04-24T00:35:33.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3382000000000001e-05 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912776+0000", 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028842700000000001 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.811331835999994, 2024-04-24T00:35:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:33.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.797508878000002, 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:33.499 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.570367693000001, 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.503 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.505 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:33.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.558578042999997, 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.514 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30704 3.1a 3:5e74e97a:::rbd_data.115e8210e51a.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.195086123999999, 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043351496000000003, 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30704 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252356+0000", 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252358+0000", 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1790000000000001e-06 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252364+0000", 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6549999999999998e-06 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252381+0000", 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6767000000000001e-05 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252390+0000", 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9959999999999999e-06 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252492+0000", 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102563 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252668+0000", 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017553499999999999 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.254545+0000", 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001877037 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295422+0000", 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040876527000000003 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295513+0000", 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1838999999999999e-05 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295614+0000", 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010027900000000001 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.295708+0000", 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4118999999999998e-05 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.564537979000001, 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.520 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:33.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.525 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33518 3.b e70/59 ondisk, result = 0)", 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.553214451000002, 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042187448000000002, 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894228+0000", 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894230+0000", 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7680000000000001e-06 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894239+0000", 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3319999999999998e-06 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894266+0000", 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049000000000001e-05 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.894281+0000", 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5245000000000001e-05 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936247+0000", 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041965833000000001 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936257+0000", 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6160000000000008e-06 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936415+0000", 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158605 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.529 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33734 3.b 3:d4ebe596:::rbd_data.115ec4f9b3a7.000000000000000e:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.928172256000003, 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041668288999999997, 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.530 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33734 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519270+0000", 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519272+0000", 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9870000000000002e-06 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519282+0000", 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0203e-05 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519310+0000", 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7880000000000001e-05 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519324+0000", 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3920000000000001e-05 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519402+0000", 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7865000000000001e-05 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.519481+0000", 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9103000000000005e-05 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.522670+0000", 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003189 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560812+0000", 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038141427999999998 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560831+0000", 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9820999999999999e-05 2024-04-24T00:35:33.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560886+0000", 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5087999999999998e-05 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:46.560938+0000", 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1993999999999997e-05 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44651 3.6 e79/59 3:6379bdb4:::rbd_data.115ec7ed39de.0000000000000000:head v 79'72, mlcod=77'70)", 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.558464888000003, 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048315631999999997, 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.888978+0000", 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7000000000000003e-07 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892459+0000", 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034802259999999999 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892475+0000", 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5998000000000002e-05 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892485+0000", 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8679999999999994e-06 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892653+0000", 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016800800000000001 2024-04-24T00:35:33.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.892721+0000", 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8953000000000002e-05 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937243+0000", 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044521666000000001 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.937293+0000", 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0043000000000002e-05 2024-04-24T00:35:33.539 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:44659 3.1d e79/53 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head v 79'105, mlcod=77'103)", 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.550983705, 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038358476000000002, 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896459+0000", 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.896460+0000", 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2399999999999997e-07 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897721+0000", 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012617049999999999 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897744+0000", 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2982000000000001e-05 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897756+0000", 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1559999999999999e-05 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897836+0000", 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0434999999999998e-05 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.897871+0000", 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5147000000000001e-05 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934696+0000", 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036824511999999997 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934817+0000", 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121311 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.6692042699999998, 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:33.544 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:35:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:35:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:33.551 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:33.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:33.553 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:33.549+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:33.654 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:33.647+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:33.680 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:33.680 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:33.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:33.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:35:33.782 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:33.779+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:33.855 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:33.849+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:33.895 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:33.895 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:33.895 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:33.896 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:33.896 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:33.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:33.958 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:33.949+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:34.056 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:34.051+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:34.112 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:34.112 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:34.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:34.112 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:34.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.399743849000004, 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.113 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:35:34.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.399645923999998, 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.119 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:34.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:35:34.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:34.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:35:34.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:35:34.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.040266391999999, 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.126 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:35:34.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:35:34.131 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.132 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.132 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.132 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30133 3.1f e66/59 ondisk, result = 0)", 2024-04-24T00:35:34.132 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.027672844999998, 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060466079999999998, 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080440+0000", 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080446+0000", 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6699999999999999e-06 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080465+0000", 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080476+0000", 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0937999999999999e-05 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140869+0000", 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060392759999999997 2024-04-24T00:35:34.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140882+0000", 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3131e-05 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140905+0000", 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2642000000000001e-05 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30705 3.2 3:45c09e0c:::rbd_data.115e8210e51a.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.855659930999998, 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063528001000000001, 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30705 2024-04-24T00:35:34.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252455+0000", 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.303e-06 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252477+0000", 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1970000000000001e-05 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252515+0000", 2024-04-24T00:35:34.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7616e-05 2024-04-24T00:35:34.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252536+0000", 2024-04-24T00:35:34.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376999999999999e-05 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252696+0000", 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160192 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252945+0000", 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024906400000000001 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.256753+0000", 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038072129999999998 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315658+0000", 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058905258000000002 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315790+0000", 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013223999999999999 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315894+0000", 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103886 2024-04-24T00:35:34.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315980+0000", 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5882000000000002e-05 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30705 3.2 e66/53 ondisk, result = 0)", 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.840447828000002, 2024-04-24T00:35:34.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048330598000000002, 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:34.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267667+0000", 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.96e-06 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267683+0000", 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6336000000000001e-05 2024-04-24T00:35:34.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267730+0000", 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7079000000000001e-05 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267757+0000", 2024-04-24T00:35:34.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6364e-05 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315724+0000", 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047967018 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315742+0000", 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7935999999999999e-05 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315995+0000", 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025290499999999999 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33506 3.c 3:3787ff34:::rbd_data.115ec4f9b3a7.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.310063786999997, 2024-04-24T00:35:34.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061037313000000003, 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33506 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798052+0000", 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798067+0000", 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5160000000000001e-05 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798107+0000", 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9946e-05 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798130+0000", 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2637e-05 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798238+0000", 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010841299999999999 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798344+0000", 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010631199999999999 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.801407+0000", 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030623600000000001 2024-04-24T00:35:34.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858616+0000", 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057209342000000003 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858879+0000", 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000262952 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858992+0000", 2024-04-24T00:35:34.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113429 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859085+0000", 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2925999999999998e-05 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33506 3.c e70/56 ondisk, result = 0)", 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.300431019999998, 2024-04-24T00:35:34.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051419185999999999, 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807682+0000", 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807687+0000", 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807704+0000", 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7424000000000001e-05 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807714+0000", 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3400000000000004e-06 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858714+0000", 2024-04-24T00:35:34.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051000248999999998 2024-04-24T00:35:34.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858732+0000", 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7995999999999998e-05 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859100+0000", 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036787400000000002 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.160 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35694 3.1b 3:dcf0eb61:::rbd_data.115ec5833ea9.00000000000000aa:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.188615169000002, 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049399446, 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35694 2024-04-24T00:35:34.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919498+0000", 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919504+0000", 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919520+0000", 2024-04-24T00:35:34.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5908000000000001e-05 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919532+0000", 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2964e-05 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919630+0000", 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7096999999999995e-05 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919734+0000", 2024-04-24T00:35:34.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104128 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.922056+0000", 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023219149999999999 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968775+0000", 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046719274999999998 2024-04-24T00:35:34.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968817+0000", 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-05 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968858+0000", 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1440999999999998e-05 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968896+0000", 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7741000000000003e-05 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.217282404999999, 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:34.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44664 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 524288~131072 [fadvise_sequential+fadvise_nocache],write 1179648~1048576 [fadvise_sequential+fadvise_nocache],write 10616832~2048 [fadvise_sequential+fadvise_nocache],write 10627072~4096 [fadvise_sequential+fadvise_nocache],write 10639360~4096 [fadvise_sequential+fadvise_nocache],write 10651648~4096 [fadvise_sequential+fadvise_nocache],write 10663936~4096 [fadvise_sequential+fadvise_nocache],write 10676224~4096 [fadvise_sequential+fadvise_nocache],write 10688512~4096 [fadvise_sequential+fadvise_nocache],write 10700800~4096 [fadvise_sequential+fadvise_nocache],write 10713088~4096 [fadvise_sequential+fadvise_nocache],write 10725376~4096 [fadvise_sequential+fadvise_nocache],write 10737664~4096 [fadvise_sequential+fadvise_nocache],write 10754048~4096 [fadvise_sequential+fadvise_nocache],write 10766336~4096 [fadvise_sequential+fadvise_nocache],write 10778624~4096 [fadvise_sequential+fadvise_nocache],write 10790912~4096 [fadvise_sequential+fadvise_nocache],write 10803200~4096 [fadvise_sequential+fadvise_nocache],write 10815488~4096 [fadvise_sequential+fadvise_nocache],write 10827776~4096 [fadvise_sequential+fadvise_nocache],write 10840064~4096 [fadvise_sequential+fadvise_nocache],write 10852352~4096 [fadvise_sequential+fadvise_nocache],write 10864640~4096 [fadvise_sequential+fadvise_nocache],write 10876928~2048 [fadvise_sequential+fadvise_nocache],write 10881024~4096 [fadvise_sequential+fadvise_nocache],write 10893312~4096 [fadvise_sequential+fadvise_nocache],write 10905600~4096 [fadvise_sequential+fadvise_nocache],write 10917888~4096 [fadvise_sequential+fadvise_nocache],write 10930176~4096 [fadvise_sequential+fadvise_nocache],write 10942464~4096 [fadvise_sequential+fadvise_nocache],write 10954752~4096 [fadvise_sequential+fadvise_nocache],write 10967040~4096 [fadvise_sequential+fadvise_nocache],write 10979328~4096 [fadvise_sequential+fadvise_nocache],write 10991616~4096 [fadvise_sequential+fadvise_nocache],write 11003904~4096 [fadvise_sequential+fadvise_nocache],write 11010048~2048 [fadvise_sequential+fadvise_nocache],write 11020288~4096 [fadvise_sequential+fadvise_nocache],write 11032576~4096 [fadvise_sequential+fadvise_nocache],write 11044864~4096 [fadvise_sequential+fadvise_nocache],write 11057152~4096 [fadvise_sequential+fadvise_nocache],write 11069440~4096 [fadvise_sequential+fadvise_nocache],write 11081728~4096 [fadvise_sequential+fadvise_nocache],write 11094016~4096 [fadvise_sequential+fadvise_nocache],write 11106304~4096 [fadvise_sequential+fadvise_nocache],write 11118592~4096 [fadvise_sequential+fadvise_nocache],write 11130880~4096 [fadvise_sequential+fadvise_nocache],write 14417920~4096 [fadvise_sequential+fadvise_nocache],write 14426112~8192 [fadvise_sequential+fadvise_nocache],write 14438400~8192 [fadvise_sequential+fadvise_nocache],write 14450688~8192 [fadvise_sequential+fadvise_nocache],write 14462976~8192 [fadvise_sequential+fadvise_nocache],write 14475264~8192 [fadvise_sequential+fadvise_nocache],write 14487552~8192 [fadvise_sequential+fadvise_nocache],write 14499840~8192 [fadvise_sequential+fadvise_nocache],write 14512128~8192 [fadvise_sequential+fadvise_nocache],write 14524416~8192 [fadvise_sequential+fadvise_nocache],write 14536704~8192 [fadvise_sequential+fadvise_nocache],write 14553088~8192 [fadvise_sequential+fadvise_nocache],write 14565376~8192 [fadvise_sequential+fadvise_nocache],write 14577664~8192 [fadvise_sequential+fadvise_nocache],write 14589952~8192 [fadvise_sequential+fadvise_nocache],write 14602240~8192 [fadvise_sequential+fadvise_nocache],write 14614528~8192 [fadvise_sequential+fadvise_nocache],write 14626816~8192 [fadvise_sequential+fadvise_nocache],write 14639104~8192 [fadvise_sequential+fadvise_nocache],write 14651392~8192 [fadvise_sequential+fadvise_nocache],write 14663680~8192 [fadvise_sequential+fadvise_nocache],write 14675968~4096 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:34.173 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.205209084000003, 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768734, 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44664 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:34.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:34.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902904+0000", 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5179999999999999e-06 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.904804+0000", 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018996390000000001 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905393+0000", 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058872600000000003 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905408+0000", 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5073e-05 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905482+0000", 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4284999999999998e-05 2024-04-24T00:35:34.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905785+0000", 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030266100000000002 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.907249+0000", 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001464899 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.932396+0000", 2024-04-24T00:35:34.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025146379 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958308+0000", 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025912418999999999 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958527+0000", 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218297 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958671+0000", 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144838 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.629522265999999, 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.180 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:35:34.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:35:34.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:35:34.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:35:34.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.187 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:35:34.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:34.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.604253660000001, 2024-04-24T00:35:34.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:34.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:35:34.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:35:34.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:35:34.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:35:34.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:35:34.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:35:34.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:35:34.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:35:34.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:35:34.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:35:34.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.193 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:67305 3.17 e89/59 ondisk, result = 0)", 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.253479584000001, 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049974565999999998, 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854633+0000", 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854637+0000", 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.946e-06 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854649+0000", 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2507000000000001e-05 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854658+0000", 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6330000000000002e-06 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904468+0000", 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049809535000000002 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904479+0000", 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-05 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904607+0000", 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012738300000000001 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:34.198 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.75104994, 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:35:34.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:35:34.204 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.718444826000001, 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:34.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:35:34.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:35:34.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:35:34.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.313870132, 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.209 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:35:34.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:35:34.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:35:34.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:35:34.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:35:34.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:35:34.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:35:34.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.215 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.256029902, 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.216 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.217 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:34.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:34.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:35:34.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:35:34.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:35:34.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:35:34.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:35:34.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:35:34.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:35:34.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:35:34.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:35:34.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:35:34.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:35:34.222 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.222 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.222 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81512 3.5 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.14489676800000001, 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061553231999999999, 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.223 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.224 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81512 2024-04-24T00:35:34.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.224 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963217+0000", 2024-04-24T00:35:34.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9439999999999999e-06 2024-04-24T00:35:34.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967964+0000", 2024-04-24T00:35:34.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047468529999999997 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967986+0000", 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1852e-05 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968000+0000", 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4426e-05 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968069+0000", 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9338000000000006e-05 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968219+0000", 2024-04-24T00:35:34.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014914999999999999 2024-04-24T00:35:34.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:34.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968363+0000", 2024-04-24T00:35:34.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144093 2024-04-24T00:35:34.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993999+0000", 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025636744999999999 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024681+0000", 2024-04-24T00:35:34.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030681452000000001 2024-04-24T00:35:34.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024722+0000", 2024-04-24T00:35:34.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1115999999999997e-05 2024-04-24T00:35:34.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024768+0000", 2024-04-24T00:35:34.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6263000000000001e-05 2024-04-24T00:35:34.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.232 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.232 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81771 3.2 3:40d81b79:::rbd_data.115e6f07caf6.0000000000000009:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:34.233 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.986665+0000", 2024-04-24T00:35:34.233 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.121446718, 2024-04-24T00:35:34.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045422974999999997, 2024-04-24T00:35:34.233 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.233 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.233 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81771 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.986665+0000", 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.986665+0000", 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.986669+0000", 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.726e-06 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.990612+0000", 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039436519999999997 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.990635+0000", 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2297999999999999e-05 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.990650+0000", 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5148e-05 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.990733+0000", 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3503000000000003e-05 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.990909+0000", 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000176093 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.991080+0000", 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017016000000000001 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.016940+0000", 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025859997999999999 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.032021+0000", 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015081699 2024-04-24T00:35:34.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.032059+0000", 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8142000000000002e-05 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.032088+0000", 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8555999999999999e-05 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.239 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.240 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:34.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:34.242 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:34.233+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:34.341 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:34.335+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:34.434 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:34.434 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:34.434 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:34.435 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:34.442 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:34.439+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:34.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:34.543 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:34.537+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:34.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:34.552 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:35:34.613 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:34.613 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:34.613 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:34.613 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:34.613 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:34.627 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:34.643 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:34.637+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:34.688 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:34.704 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:34.704 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:34.720 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:34.721 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:34.721 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-04-24T00:35:34.737 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:34.744 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:34.739+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:34.755 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:34.755 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:34.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:34.755 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:34.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.755 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10771 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:34.755 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:34.755 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.467349245999998, 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016835930999999998, 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10771 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:34.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282814+0000", 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282815+0000", 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8700000000000005e-07 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282822+0000", 2024-04-24T00:35:34.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2860000000000002e-06 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282831+0000", 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3440000000000007e-06 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282839+0000", 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0960000000000001e-06 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.291986+0000", 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091468750000000005 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292133+0000", 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147322 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.292779+0000", 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064593400000000003 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298553+0000", 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057735989999999999 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299408+0000", 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085489300000000002 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299560+0000", 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015224999999999999 2024-04-24T00:35:34.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299650+0000", 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9544999999999997e-05 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10772 3.a 3:56409d54:::rbd_object_map.115e380d621c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.467313232999999, 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017111035, 2024-04-24T00:35:34.761 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10772 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282850+0000", 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282851+0000", 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9699999999999995e-07 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282854+0000", 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8759999999999998e-06 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282864+0000", 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.282871+0000", 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6699999999999994e-06 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.293991+0000", 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011119386 2024-04-24T00:35:34.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294123+0000", 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013204199999999999 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.294703+0000", 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058024300000000003 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.298600+0000", 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038973689999999999 2024-04-24T00:35:34.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299760+0000", 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001159881 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299894+0000", 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000133697 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:14.299961+0000", 2024-04-24T00:35:34.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6804000000000006e-05 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.217908839000003, 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:35:34.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.183871785999997, 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:35:34.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:35:34.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.593145821999997, 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:35:34.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:35:34.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:35:34.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:35:34.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:34.782 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 53.583889697000004, 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:35:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 44.068757789999999, 2024-04-24T00:35:34.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:35:34.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:35:34.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:35:34.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 44.041307828999997, 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:35:34.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:35:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:35:34.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:42868 3.1e 3:781b6cfc:::rbd_data.115e56ad4d37.000000000000003c:snapdir [list-snaps] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.996606624000002, 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021882233000000001, 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.796 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 42868 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753558+0000", 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6869999999999999e-06 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753573+0000", 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.446e-05 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753615+0000", 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.227e-05 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753633+0000", 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8240000000000002e-05 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.754227+0000", 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059412099999999997 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.775439+0000", 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021211455000000001 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.799 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72671 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.098326168, 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017794547000000001, 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72671 2024-04-24T00:35:34.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651841+0000", 2024-04-24T00:35:34.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651856+0000", 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5549999999999999e-05 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651907+0000", 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1187000000000003e-05 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651933+0000", 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5550000000000001e-05 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652300+0000", 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036690699999999998 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652318+0000", 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8315999999999999e-05 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669396+0000", 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017078233000000002 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669631+0000", 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000235028 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72679 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:34.804 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.098149056, 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019600632999999999, 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72679 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652016+0000", 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652025+0000", 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8729999999999996e-06 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652047+0000", 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2957000000000001e-05 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652065+0000", 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7289999999999999e-05 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652773+0000", 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070851400000000004 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652791+0000", 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7649999999999999e-05 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671513+0000", 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018722486 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671615+0000", 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010119 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72730 3.a 3:5760bc47:::rbd_data.115efb339e04.000000000000004d:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653326+0000", 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.096836561, 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016881816000000001, 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.809 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72730 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653326+0000", 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653326+0000", 2024-04-24T00:35:34.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653327+0000", 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8000000000000004e-07 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653332+0000", 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9239999999999998e-06 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653346+0000", 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3762e-05 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653357+0000", 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0949e-05 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654723+0000", 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013659589999999999 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:34.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654745+0000", 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2155e-05 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669983+0000", 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015237812 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670208+0000", 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022537499999999999 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72737 3.a 3:5752c519:::rbd_data.115efb339e04.0000000000000056:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653372+0000", 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.096791257, 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017125208999999999, 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72737 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.814 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653372+0000", 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653372+0000", 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653373+0000", 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8700000000000004e-07 2024-04-24T00:35:34.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653377+0000", 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5020000000000003e-06 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653391+0000", 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3417999999999999e-05 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653417+0000", 2024-04-24T00:35:34.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6452e-05 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654797+0000", 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013801169999999999 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654807+0000", 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9569999999999997e-06 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670279+0000", 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015471468 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670497+0000", 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021840799999999999 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72759 3.a 3:541a2ac3:::rbd_data.115efb339e04.000000000000006f:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:34.818 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.096540258999999, 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017161025999999999, 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72759 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653624+0000", 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653629+0000", 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1529999999999996e-06 2024-04-24T00:35:34.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653660+0000", 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0623999999999997e-05 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653718+0000", 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8514999999999998e-05 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654870+0000", 2024-04-24T00:35:34.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001151909 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654881+0000", 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.118e-05 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670564+0000", 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015682413999999999 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670784+0000", 2024-04-24T00:35:34.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022022799999999999 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72761 3.a 3:50a1eac1:::rbd_data.115efb339e04.0000000000000072:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.096337918, 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017246102999999999, 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72761 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653826+0000", 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9699999999999994e-07 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653843+0000", 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7240000000000001e-05 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653857+0000", 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3823e-05 2024-04-24T00:35:34.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653868+0000", 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1055e-05 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654965+0000", 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001097085 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654987+0000", 2024-04-24T00:35:34.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1759999999999998e-05 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670852+0000", 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015864494999999999 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671071+0000", 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021964799999999999 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72776 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.096260268, 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017442424000000002, 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72776 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653904+0000", 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1200000000000001e-06 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653914+0000", 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7669999999999998e-06 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653928+0000", 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4165000000000001e-05 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653940+0000", 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-05 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655028+0000", 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010878579999999999 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655033+0000", 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9339999999999997e-06 2024-04-24T00:35:34.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671139+0000", 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016105922000000002 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671345+0000", 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020662799999999999 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72783 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.096206693999999, 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017776166999999999, 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72783 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653957+0000", 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1699999999999997e-07 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653962+0000", 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6079999999999998e-06 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653975+0000", 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2974e-05 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653986+0000", 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0851e-05 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654503+0000", 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051748099999999997 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654523+0000", 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0032999999999999e-05 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671644+0000", 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017121032000000001 2024-04-24T00:35:34.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671733+0000", 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8270999999999996e-05 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:81511 3.d e111/106 3:b780edbf:::rbd_data.115e6f07caf6.0000000000000006:head v 111'118, mlcod=0'0)", 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.78631151099999996, 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029826869999999998, 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.837 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.65e-07 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966954+0000", 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031022379999999998 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966976+0000", 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1565000000000001e-05 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966988+0000", 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2428e-05 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967105+0000", 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011725799999999999 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967132+0000", 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6863999999999999e-05 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993615+0000", 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026482946 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993678+0000", 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3206000000000004e-05 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84130 3.11 e111/106 3:8ac0652c:::rbd_data.115e6f07caf6.000000000000000b:head v 111'344, mlcod=108'342)", 2024-04-24T00:35:34.841 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.57976848400000003, 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037411427999999997, 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0699999999999999e-07 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171887+0000", 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00149255 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171909+0000", 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1270000000000001e-05 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171918+0000", 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6139999999999998e-06 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172147+0000", 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022887100000000001 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172341+0000", 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019398700000000001 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207639+0000", 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035298445999999997 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207806+0000", 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016648300000000001 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84132 3.14 e111/106 3:2c6d4ac0:::rbd_data.115e6f07caf6.000000000000000d:head v 111'167, mlcod=0'0)", 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.56993462399999995, 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029744291999999999, 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:34.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180230+0000", 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8029999999999999e-06 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.184974+0000", 2024-04-24T00:35:34.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047440540000000002 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185003+0000", 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8860000000000002e-05 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185021+0000", 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7473999999999999e-05 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:34.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185169+0000", 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148305 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185196+0000", 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7444e-05 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209925+0000", 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024728208000000002 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:34.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209973+0000", 2024-04-24T00:35:34.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8143999999999998e-05 2024-04-24T00:35:34.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.850 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:34.850 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:34.850 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:34.851 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:34.852 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:34.852 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:34.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:34.853 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:34.847+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:34.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:34.853 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:34.854 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:34.854 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:34.952 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:34.945+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:35.019 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:35.019 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:35.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:35.053 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:35.045+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:35.154 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:35.147+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:35.158 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:35.158 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:35.172 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:35.254 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:35.249+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:35.339 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:35.339 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:35.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:35.355 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:35.351+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:35.455 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:35.451+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:35.489 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:35.489 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:35.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:35.556 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:35.551+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:35.652 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:35.653 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:35.657 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:35.649+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:35.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:35.758 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:35.755+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:35.832 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:35.833 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:35.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:35.879 INFO:teuthology.orchestra.run.smithi064.stdout:3176 trunc from 0xdce12b2 to 0x60bf96c 2024-04-24T00:35:35.879 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:35.873+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:35.925 INFO:teuthology.orchestra.run.smithi064.stdout:3177 read 0x2cd41c8 thru 0x2ce1fb4 (0xdded bytes) 2024-04-24T00:35:35.947 INFO:teuthology.orchestra.run.smithi064.stdout:3178 write 0xedb8145 thru 0xedc1a27 (0x98e3 bytes) 2024-04-24T00:35:35.960 INFO:teuthology.orchestra.run.smithi064.stdout:3180 write 0x48daa99 thru 0x48dd0dc (0x2644 bytes) 2024-04-24T00:35:35.960 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:35.955+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:35.964 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:35.964 INFO:teuthology.orchestra.run.smithi064.stdout:3182 read 0xc5807b1 thru 0xc590266 (0xfab6 bytes) 2024-04-24T00:35:35.964 INFO:teuthology.orchestra.run.smithi064.stdout:3185 read 0x3497150 thru 0x34a5b0d (0xe9be bytes) 2024-04-24T00:35:35.966 INFO:teuthology.orchestra.run.smithi064.stdout:3186 read 0xa05c5fc thru 0xa05ecca (0x26cf bytes) 2024-04-24T00:35:35.968 INFO:teuthology.orchestra.run.smithi064.stdout:3187 trunc from 0xedc1a28 to 0xd408fc7 2024-04-24T00:35:35.977 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:35.977 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:35.987 INFO:teuthology.orchestra.run.smithi064.stdout:3188 write 0xd06f64e thru 0xd078914 (0x92c7 bytes) 2024-04-24T00:35:35.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:35.999 INFO:teuthology.orchestra.run.smithi064.stdout:3190 write 0x2289b12 thru 0x2290511 (0x6a00 bytes) 2024-04-24T00:35:36.004 INFO:teuthology.orchestra.run.smithi064.stdout:3191 read 0xa833303 thru 0xa83c04a (0x8d48 bytes) 2024-04-24T00:35:36.005 INFO:teuthology.orchestra.run.smithi064.stdout:3192 write 0x85b0e94 thru 0x85b4541 (0x36ae bytes) 2024-04-24T00:35:36.013 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:36.013 INFO:teuthology.orchestra.run.smithi064.stdout:3194 read 0x13e8dea thru 0x13ed6b9 (0x48d0 bytes) 2024-04-24T00:35:36.015 INFO:teuthology.orchestra.run.smithi064.stdout:3195 writesame 0x5b087c1 thru 0x5b0b26f data_size 0x7(0x2aaf bytes) 2024-04-24T00:35:36.034 INFO:teuthology.orchestra.run.smithi064.stdout:3197 trunc from 0xd408fc7 to 0x1f5d988 2024-04-24T00:35:36.061 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:36.055+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:36.073 INFO:teuthology.orchestra.run.smithi064.stdout:3198 read 0x1b521a thru 0x1c3c0b (0xe9f2 bytes) 2024-04-24T00:35:36.074 INFO:teuthology.orchestra.run.smithi064.stdout:3199 read 0x19c2c96 thru 0x19c6004 (0x336f bytes) 2024-04-24T00:35:36.076 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:36.076 INFO:teuthology.orchestra.run.smithi064.stdout:3201 read 0x84e27c thru 0x85d0d2 (0xee57 bytes) 2024-04-24T00:35:36.091 INFO:teuthology.orchestra.run.smithi064.stdout:3202 writesame 0x63e068a thru 0x63e3926 data_size 0x7(0x329d bytes) 2024-04-24T00:35:36.091 INFO:teuthology.orchestra.run.smithi064.stdout:3203 read 0x21364f2 thru 0x213b37c (0x4e8b bytes) 2024-04-24T00:35:36.091 INFO:teuthology.orchestra.run.smithi064.stdout:3204 write 0x40a6c05 thru 0x40ae5e6 (0x79e2 bytes) 2024-04-24T00:35:36.106 INFO:teuthology.orchestra.run.smithi064.stdout:3208 writesame 0x574fac2 thru 0x5758ee2 data_size 0x1(0x9421 bytes) 2024-04-24T00:35:36.121 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:36.121 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:36.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:36.162 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:36.159+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:36.203 INFO:teuthology.orchestra.run.smithi064.stdout:3209 write 0x7fbd59e thru 0x7fc2494 (0x4ef7 bytes) 2024-04-24T00:35:36.223 INFO:teuthology.orchestra.run.smithi064.stdout:3211 write 0x9621aba thru 0x96243be (0x2905 bytes) 2024-04-24T00:35:36.233 INFO:teuthology.orchestra.run.smithi064.stdout:3213 read 0x6c70ed7 thru 0x6c7f559 (0xe683 bytes) 2024-04-24T00:35:36.238 INFO:teuthology.orchestra.run.smithi064.stdout:3214 trunc from 0x96243bf to 0xc03712d 2024-04-24T00:35:36.257 INFO:teuthology.orchestra.run.smithi064.stdout:3215 trunc from 0xc03712d to 0x3c1ed62 2024-04-24T00:35:36.263 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:36.259+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:36.305 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:36.305 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:36.316 INFO:teuthology.orchestra.run.smithi064.stdout:3216 writesame 0xc77c29d thru 0xc77c604 data_size 0x1(0x368 bytes) 2024-04-24T00:35:36.316 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:36.316 INFO:teuthology.orchestra.run.smithi064.stdout:3219 punch from 0x70fac3b to 0x7101b7f, (0x6f44 bytes) 2024-04-24T00:35:36.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:36.331 INFO:teuthology.orchestra.run.smithi064.stdout:3220 trunc from 0xc77c605 to 0x4edcd50 2024-04-24T00:35:36.346 INFO:teuthology.orchestra.run.smithi064.stdout:3221 read 0x2525bc2 thru 0x252cd3f (0x717e bytes) 2024-04-24T00:35:36.347 INFO:teuthology.orchestra.run.smithi064.stdout:3222 write 0x3bd5830 thru 0x3bd681f (0xff0 bytes) 2024-04-24T00:35:36.361 INFO:teuthology.orchestra.run.smithi064.stdout:3223 trunc from 0x4edcd50 to 0xdbe6f94 2024-04-24T00:35:36.363 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:36.359+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:36.370 INFO:teuthology.orchestra.run.smithi064.stdout:3227 writesame 0x38a5dcc thru 0x38a6364 data_size 0x1(0x599 bytes) 2024-04-24T00:35:36.376 INFO:teuthology.orchestra.run.smithi064.stdout:3228 punch from 0xc2b16ca to 0xc2b79b2, (0x62e8 bytes) 2024-04-24T00:35:36.376 INFO:teuthology.orchestra.run.smithi064.stdout:3229 read 0xc0d7524 thru 0xc0dc8a9 (0x5386 bytes) 2024-04-24T00:35:36.376 INFO:teuthology.orchestra.run.smithi064.stdout:3230 write 0x8aad5e9 thru 0x8abcf09 (0xf921 bytes) 2024-04-24T00:35:36.398 INFO:teuthology.orchestra.run.smithi064.stdout:3232 read 0x3dd01d2 thru 0x3ddf830 (0xf65f bytes) 2024-04-24T00:35:36.400 INFO:teuthology.orchestra.run.smithi064.stdout:3233 read 0x5fb96ce thru 0x5fc348d (0x9dc0 bytes) 2024-04-24T00:35:36.400 INFO:teuthology.orchestra.run.smithi064.stdout:3236 punch from 0x9ff8cc4 to 0xa0033a6, (0xa6e2 bytes) 2024-04-24T00:35:36.400 INFO:teuthology.orchestra.run.smithi064.stdout:3237 write 0xcd93f64 thru 0xcd98a9d (0x4b3a bytes) 2024-04-24T00:35:36.417 INFO:teuthology.orchestra.run.smithi064.stdout:3238 trunc from 0xdbe6f94 to 0xc2c3e49 2024-04-24T00:35:36.433 INFO:teuthology.orchestra.run.smithi064.stdout:3241 writesame 0x10f3a54 thru 0x10ffa71 data_size 0x15(0xc01e bytes) 2024-04-24T00:35:36.439 INFO:teuthology.orchestra.run.smithi064.stdout:3243 write 0x3f0121f thru 0x3f0f668 (0xe44a bytes) 2024-04-24T00:35:36.456 INFO:teuthology.orchestra.run.smithi064.stdout:3245 punch from 0x22e084a to 0x22ea8ab, (0xa061 bytes) 2024-04-24T00:35:36.457 INFO:teuthology.orchestra.run.smithi064.stdout:3246 read 0xbc0aeee thru 0xbc17186 (0xc299 bytes) 2024-04-24T00:35:36.457 INFO:teuthology.orchestra.run.smithi064.stdout:3248 read 0x90569c thru 0x913c04 (0xe569 bytes) 2024-04-24T00:35:36.458 INFO:teuthology.orchestra.run.smithi064.stdout:3249 punch from 0x36274 to 0x44b03, (0xe88f bytes) 2024-04-24T00:35:36.459 INFO:teuthology.orchestra.run.smithi064.stdout:3251 clone 21 order 25 su 8388608 sc 4 2024-04-24T00:35:36.464 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:36.457+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:36.468 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:36.468 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:36.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:36.564 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:36.559+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:36.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:35:36.661 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:36.661 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:36.664 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:36.657+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:36.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:36.744 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:35:36.744 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:35:36.744 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-24T00:35:36.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-24T00:35:36.766 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:36.759+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:36.838 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:36.838 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:36.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:36.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:36.866 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:36.863+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:36.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-24T00:35:36.967 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:36.963+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:36.984 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:36.984 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:36.999 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:36.999 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:37.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:37.032 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:37.033 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:37.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:37.065 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:37.067 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:37.063+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:37.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:37.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:37.116 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:37.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:37.140 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:37.140 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:37.151 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:37.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:37.152 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:37.152 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:37.152 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:37.162 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:37.162 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:37.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:37.169 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:37.161+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:37.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-24T00:35:37.183 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:37.184 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:37.269 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:37.263+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:37.301 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.086817581000005, 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.302 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:35:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.086466666000007, 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:35:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.079578346000005, 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:35:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.821296719999999, 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:35:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.811020304000003, 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3490695009999998, 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:35:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3366612350000002, 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:35:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3366294810000001, 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:35:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:35:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81726 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 1523712~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965277+0000", 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3333009100000002, 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040967534, 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81726 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965277+0000", 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965277+0000", 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965278+0000", 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5199999999999998e-07 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965280+0000", 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.593e-06 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965287+0000", 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4119999999999998e-06 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965292+0000", 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1309999999999997e-06 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006043+0000", 2024-04-24T00:35:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040751300999999997 2024-04-24T00:35:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006130+0000", 2024-04-24T00:35:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6761000000000005e-05 2024-04-24T00:35:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006245+0000", 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114984 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81727 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 1671168~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965298+0000", 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3332799770000001, 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041195462000000002, 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81727 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965298+0000", 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965298+0000", 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965299+0000", 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965301+0000", 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5440000000000001e-06 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965307+0000", 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4060000000000004e-06 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965312+0000", 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8500000000000002e-06 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006280+0000", 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040968063999999998 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006376+0000", 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6187999999999995e-05 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006494+0000", 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011797 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81728 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 1818624~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965318+0000", 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3332604350000001, 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041364740999999997, 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81728 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965318+0000", 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965318+0000", 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965318+0000", 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.84e-07 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965321+0000", 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4779999999999998e-06 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965327+0000", 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8250000000000001e-06 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965332+0000", 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1479999999999997e-06 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006530+0000", 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041197833000000003 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006600+0000", 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0611000000000005e-05 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006683+0000", 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2461999999999998e-05 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81729 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2097152~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965338+0000", 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3332406190000001, 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041524683, 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81729 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965338+0000", 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965338+0000", 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965338+0000", 2024-04-24T00:35:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8099999999999998e-07 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965341+0000", 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4899999999999999e-06 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965346+0000", 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.84e-06 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965351+0000", 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7339999999999999e-06 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006710+0000", 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041359360999999997 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006780+0000", 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9855000000000001e-05 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006862+0000", 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2021999999999995e-05 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81730 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2244608~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965358+0000", 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3332207629999999, 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041687266000000001, 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81730 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965358+0000", 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965358+0000", 2024-04-24T00:35:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965358+0000", 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3200000000000005e-07 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965361+0000", 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965367+0000", 2024-04-24T00:35:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1e-06 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965372+0000", 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0010000000000001e-06 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006895+0000", 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041523453000000002 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.006965+0000", 2024-04-24T00:35:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9939999999999998e-05 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007045+0000", 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9720000000000002e-05 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81731 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2392064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965378+0000", 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3332007089999998, 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041855058000000001, 2024-04-24T00:35:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81731 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965378+0000", 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965378+0000", 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965378+0000", 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.72e-07 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965380+0000", 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4169999999999999e-06 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965386+0000", 2024-04-24T00:35:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9510000000000003e-06 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965391+0000", 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8579999999999999e-06 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007071+0000", 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041679573999999997 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007142+0000", 2024-04-24T00:35:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1027999999999998e-05 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007233+0000", 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0858000000000003e-05 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81732 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2539520~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3331806249999998, 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042008566999999997, 2024-04-24T00:35:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81732 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965400+0000", 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4150000000000002e-06 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965407+0000", 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0920000000000003e-06 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965411+0000", 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7099999999999998e-06 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007262+0000", 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041850738999999998 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007328+0000", 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6375000000000001e-05 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007406+0000", 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7844999999999998e-05 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81733 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2818048~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3331250880000001, 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042136290999999999, 2024-04-24T00:35:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81733 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965454+0000", 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965457+0000", 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3799999999999998e-06 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965466+0000", 2024-04-24T00:35:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5280000000000003e-06 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965472+0000", 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4670000000000001e-06 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007435+0000", 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041963037000000002 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007502+0000", 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6727e-05 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007590+0000", 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7614999999999994e-05 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3330969850000001, 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.3230354800000002, 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:35:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:35:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:35:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.1090925729999999, 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.0990806289999999, 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:35:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:35:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:37.391 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:37.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:37.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-24T00:35:37.394 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:37.385+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:37.493 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:37.487+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:37.527 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:37.527 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:37.527 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:37.528 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:37.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:37.566 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:35:37.593 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:37.589+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:37.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-04-24T00:35:37.694 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:37.691+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:37.757 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:37.757 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:37.757 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:37.757 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:37.757 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:37.772 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:37.814 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:37.811+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:37.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-24T00:35:37.896 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:37.891+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:37.930 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:37.930 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.713454560000002, 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:37.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:37.936 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.697925311000006, 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:37.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.069201991, 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:37.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:37.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:37.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.291668865999995, 2024-04-24T00:35:37.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:37.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.277845908000003, 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:37.951 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:37.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:37.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.050704723000003, 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:37.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:37.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:37.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:37.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:37.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.038915072999998, 2024-04-24T00:35:37.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.044875009000002, 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.965 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:37.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.970 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.1495412999999992, 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:35:37.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:35:37.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.0117695180000004, 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.977 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.982 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.9742872, 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:35:37.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:35:37.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:35:37.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:81510 3.6 e111/59 3:6416f24e:::rbd_data.115e6f07caf6.0000000000000004:head v 111'174, mlcod=108'172)", 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.975662+0000", 2024-04-24T00:35:37.988 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.9521177839999999, 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051816996999999997, 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.975662+0000", 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.975662+0000", 2024-04-24T00:35:37.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.975664+0000", 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0150000000000002e-06 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.981634+0000", 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005970365 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.981659+0000", 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4564999999999999e-05 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.981672+0000", 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3266e-05 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.994361+0000", 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012688708999999999 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.994385+0000", 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4603000000000001e-05 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027449+0000", 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033063412 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027479+0000", 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0062000000000001e-05 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.992 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.9449582379999999, 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:35:37.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:37.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84125 3.9 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.151724+0000", 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.776055457, 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064059736000000006, 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.001 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84125 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.151724+0000", 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.151724+0000", 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.151728+0000", 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8099999999999999e-06 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.155992+0000", 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004264164 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156017+0000", 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5103e-05 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156033+0000", 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6342e-05 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156157+0000", 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012311900000000001 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156281+0000", 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012401600000000001 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156381+0000", 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100079 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.175498+0000", 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019117173000000001 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.215721+0000", 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040223354000000003 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.215756+0000", 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483999999999997e-05 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.215784+0000", 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8092000000000001e-05 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.006 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84130 3.11 3:8ac0652c:::rbd_data.115e6f07caf6.000000000000000b:head [write 0~4030464 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.165649+0000", 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7621302679999999, 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053811444999999999, 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84130 2024-04-24T00:35:38.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.165649+0000", 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.165649+0000", 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.165651+0000", 2024-04-24T00:35:38.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1509999999999998e-06 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.169750+0000", 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040984439999999997 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.169774+0000", 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4247e-05 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.169786+0000", 2024-04-24T00:35:38.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2354000000000001e-05 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.169868+0000", 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2076999999999994e-05 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170049+0000", 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018052699999999999 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170219+0000", 2024-04-24T00:35:38.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017027700000000001 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.215809+0000", 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045589835000000002 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219354+0000", 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035447389999999999 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219412+0000", 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7655999999999998e-05 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219461+0000", 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9138000000000003e-05 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7476963350000001, 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:35:38.012 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:35:38.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:35:38.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:35:38.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:35:38.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.31194133600000001, 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.023 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.3020833, 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:35:38.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.28173369199999998, 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.029 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.034 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87736 3.a e115/21 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head v 115'512, mlcod=111'510)", 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.652456+0000", 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.27532300599999998, 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061811680000000001, 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652456+0000", 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652456+0000", 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652459+0000", 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.656878+0000", 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044193660000000001 2024-04-24T00:35:38.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.656908+0000", 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9828e-05 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.656925+0000", 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7674000000000001e-05 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.657043+0000", 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117351 2024-04-24T00:35:38.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.657074+0000", 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1157e-05 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.714206+0000", 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057132170000000003 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.714268+0000", 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2033999999999996e-05 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.039 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:38.041 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:38.033+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:38.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:38.141 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:38.135+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:38.209 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:38.209 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:38.209 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:38.209 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:38.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:38.242 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:38.237+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:38.343 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:38.337+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:38.390 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:38.391 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:38.391 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:38.391 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:38.391 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:38.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:38.450 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:38.447+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.814542896999995, 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:35:38.528 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:35:38.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:38.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.814444972000004, 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.455065439999998, 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.540 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:35:38.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:35:38.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.545 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30133 3.1f e66/59 ondisk, result = 0)", 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.442471892999997, 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060466079999999998, 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080440+0000", 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080446+0000", 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6699999999999999e-06 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080465+0000", 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080476+0000", 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0937999999999999e-05 2024-04-24T00:35:38.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140869+0000", 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060392759999999997 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140882+0000", 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3131e-05 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140905+0000", 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2642000000000001e-05 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30705 3.2 3:45c09e0c:::rbd_data.115e8210e51a.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.270458978999997, 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063528001000000001, 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.550 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30705 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:38.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252455+0000", 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.303e-06 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252477+0000", 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1970000000000001e-05 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252515+0000", 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7616e-05 2024-04-24T00:35:38.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252536+0000", 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376999999999999e-05 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252696+0000", 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160192 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252945+0000", 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024906400000000001 2024-04-24T00:35:38.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.256753+0000", 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038072129999999998 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315658+0000", 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058905258000000002 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315790+0000", 2024-04-24T00:35:38.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013223999999999999 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315894+0000", 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103886 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315980+0000", 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5882000000000002e-05 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30705 3.2 e66/53 ondisk, result = 0)", 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.255246876000001, 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048330598000000002, 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267667+0000", 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.96e-06 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267683+0000", 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6336000000000001e-05 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267730+0000", 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7079000000000001e-05 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267757+0000", 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6364e-05 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315724+0000", 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047967018 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315742+0000", 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7935999999999999e-05 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315995+0000", 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025290499999999999 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33506 3.c 3:3787ff34:::rbd_data.115ec4f9b3a7.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:38.559 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.724862835000003, 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061037313000000003, 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33506 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:38.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798052+0000", 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798067+0000", 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5160000000000001e-05 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798107+0000", 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9946e-05 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798130+0000", 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2637e-05 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798238+0000", 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010841299999999999 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798344+0000", 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010631199999999999 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.801407+0000", 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030623600000000001 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858616+0000", 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057209342000000003 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858879+0000", 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000262952 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858992+0000", 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113429 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859085+0000", 2024-04-24T00:35:38.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2925999999999998e-05 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33506 3.c e70/56 ondisk, result = 0)", 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.715230067999997, 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051419185999999999, 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807682+0000", 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807687+0000", 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807704+0000", 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7424000000000001e-05 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807714+0000", 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3400000000000004e-06 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858714+0000", 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051000248999999998 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858732+0000", 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7995999999999998e-05 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859100+0000", 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036787400000000002 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35694 3.1b 3:dcf0eb61:::rbd_data.115ec5833ea9.00000000000000aa:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.603414217000001, 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049399446, 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35694 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.569 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919498+0000", 2024-04-24T00:35:38.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919504+0000", 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919520+0000", 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5908000000000001e-05 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919532+0000", 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2964e-05 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919630+0000", 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7096999999999995e-05 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919734+0000", 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104128 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.922056+0000", 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023219149999999999 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968775+0000", 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046719274999999998 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968817+0000", 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-05 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968858+0000", 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1440999999999998e-05 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968896+0000", 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7741000000000003e-05 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.632081452999998, 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.574 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:35:38.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:35:38.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44664 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 524288~131072 [fadvise_sequential+fadvise_nocache],write 1179648~1048576 [fadvise_sequential+fadvise_nocache],write 10616832~2048 [fadvise_sequential+fadvise_nocache],write 10627072~4096 [fadvise_sequential+fadvise_nocache],write 10639360~4096 [fadvise_sequential+fadvise_nocache],write 10651648~4096 [fadvise_sequential+fadvise_nocache],write 10663936~4096 [fadvise_sequential+fadvise_nocache],write 10676224~4096 [fadvise_sequential+fadvise_nocache],write 10688512~4096 [fadvise_sequential+fadvise_nocache],write 10700800~4096 [fadvise_sequential+fadvise_nocache],write 10713088~4096 [fadvise_sequential+fadvise_nocache],write 10725376~4096 [fadvise_sequential+fadvise_nocache],write 10737664~4096 [fadvise_sequential+fadvise_nocache],write 10754048~4096 [fadvise_sequential+fadvise_nocache],write 10766336~4096 [fadvise_sequential+fadvise_nocache],write 10778624~4096 [fadvise_sequential+fadvise_nocache],write 10790912~4096 [fadvise_sequential+fadvise_nocache],write 10803200~4096 [fadvise_sequential+fadvise_nocache],write 10815488~4096 [fadvise_sequential+fadvise_nocache],write 10827776~4096 [fadvise_sequential+fadvise_nocache],write 10840064~4096 [fadvise_sequential+fadvise_nocache],write 10852352~4096 [fadvise_sequential+fadvise_nocache],write 10864640~4096 [fadvise_sequential+fadvise_nocache],write 10876928~2048 [fadvise_sequential+fadvise_nocache],write 10881024~4096 [fadvise_sequential+fadvise_nocache],write 10893312~4096 [fadvise_sequential+fadvise_nocache],write 10905600~4096 [fadvise_sequential+fadvise_nocache],write 10917888~4096 [fadvise_sequential+fadvise_nocache],write 10930176~4096 [fadvise_sequential+fadvise_nocache],write 10942464~4096 [fadvise_sequential+fadvise_nocache],write 10954752~4096 [fadvise_sequential+fadvise_nocache],write 10967040~4096 [fadvise_sequential+fadvise_nocache],write 10979328~4096 [fadvise_sequential+fadvise_nocache],write 10991616~4096 [fadvise_sequential+fadvise_nocache],write 11003904~4096 [fadvise_sequential+fadvise_nocache],write 11010048~2048 [fadvise_sequential+fadvise_nocache],write 11020288~4096 [fadvise_sequential+fadvise_nocache],write 11032576~4096 [fadvise_sequential+fadvise_nocache],write 11044864~4096 [fadvise_sequential+fadvise_nocache],write 11057152~4096 [fadvise_sequential+fadvise_nocache],write 11069440~4096 [fadvise_sequential+fadvise_nocache],write 11081728~4096 [fadvise_sequential+fadvise_nocache],write 11094016~4096 [fadvise_sequential+fadvise_nocache],write 11106304~4096 [fadvise_sequential+fadvise_nocache],write 11118592~4096 [fadvise_sequential+fadvise_nocache],write 11130880~4096 [fadvise_sequential+fadvise_nocache],write 14417920~4096 [fadvise_sequential+fadvise_nocache],write 14426112~8192 [fadvise_sequential+fadvise_nocache],write 14438400~8192 [fadvise_sequential+fadvise_nocache],write 14450688~8192 [fadvise_sequential+fadvise_nocache],write 14462976~8192 [fadvise_sequential+fadvise_nocache],write 14475264~8192 [fadvise_sequential+fadvise_nocache],write 14487552~8192 [fadvise_sequential+fadvise_nocache],write 14499840~8192 [fadvise_sequential+fadvise_nocache],write 14512128~8192 [fadvise_sequential+fadvise_nocache],write 14524416~8192 [fadvise_sequential+fadvise_nocache],write 14536704~8192 [fadvise_sequential+fadvise_nocache],write 14553088~8192 [fadvise_sequential+fadvise_nocache],write 14565376~8192 [fadvise_sequential+fadvise_nocache],write 14577664~8192 [fadvise_sequential+fadvise_nocache],write 14589952~8192 [fadvise_sequential+fadvise_nocache],write 14602240~8192 [fadvise_sequential+fadvise_nocache],write 14614528~8192 [fadvise_sequential+fadvise_nocache],write 14626816~8192 [fadvise_sequential+fadvise_nocache],write 14639104~8192 [fadvise_sequential+fadvise_nocache],write 14651392~8192 [fadvise_sequential+fadvise_nocache],write 14663680~8192 [fadvise_sequential+fadvise_nocache],write 14675968~4096 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.620008132000002, 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768734, 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44664 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902904+0000", 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5179999999999999e-06 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.904804+0000", 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018996390000000001 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905393+0000", 2024-04-24T00:35:38.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058872600000000003 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905408+0000", 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5073e-05 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905482+0000", 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4284999999999998e-05 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905785+0000", 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030266100000000002 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.907249+0000", 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001464899 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.932396+0000", 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025146379 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958308+0000", 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025912418999999999 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958527+0000", 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218297 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958671+0000", 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144838 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.044321314000001, 2024-04-24T00:35:38.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.590 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.019052707999997, 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:38.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.594 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:67305 3.17 e89/59 ondisk, result = 0)", 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.668278632, 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049974565999999998, 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:38.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854633+0000", 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854637+0000", 2024-04-24T00:35:38.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.946e-06 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854649+0000", 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2507000000000001e-05 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854658+0000", 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6330000000000002e-06 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904468+0000", 2024-04-24T00:35:38.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049809535000000002 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904479+0000", 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-05 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904607+0000", 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012738300000000001 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.598 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.165848988, 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:35:38.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:35:38.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:38.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:35:38.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.133243874000001, 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:35:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.728669180000001, 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.608 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:35:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.670828950000001, 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:35:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81512 3.5 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5596958159999996, 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061553231999999999, 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81512 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963217+0000", 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9439999999999999e-06 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967964+0000", 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047468529999999997 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967986+0000", 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1852e-05 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968000+0000", 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4426e-05 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968069+0000", 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9338000000000006e-05 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968219+0000", 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014914999999999999 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968363+0000", 2024-04-24T00:35:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144093 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993999+0000", 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025636744999999999 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024681+0000", 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030681452000000001 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024722+0000", 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1115999999999997e-05 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024768+0000", 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6263000000000001e-05 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84133 3.1f 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head [write 0~4030464 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.3504454639999999, 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060722168, 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:38.623 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84133 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:38.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172468+0000", 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2869999999999999e-06 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180251+0000", 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0077834189999999998 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180276+0000", 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4932000000000001e-05 2024-04-24T00:35:38.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180288+0000", 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2224e-05 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180378+0000", 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9401999999999995e-05 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180542+0000", 2024-04-24T00:35:38.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016484599999999999 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180700+0000", 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015754300000000001 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.208480+0000", 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027780143 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233097+0000", 2024-04-24T00:35:38.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02461675 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233151+0000", 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4593999999999999e-05 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233187+0000", 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6028000000000001e-05 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:38.628 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:38.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:38.630 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:38.625+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:38.731 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:38.725+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:38.802 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-04-24T00:35:38.805 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:38.806 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:38.806 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:38.806 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:38.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:38.866 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:38.863+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:38.933 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:38.927+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:38.938 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:38.938 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:38.938 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:38.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:39.034 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:39.029+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:39.117 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:39.117 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:39.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:39.117 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.581932979000001, 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:35:39.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:35:39.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:35:39.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.547895925999995, 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.123 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.126 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.957169962000002, 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.127 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:35:39.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:35:39.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.947913837000002, 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:35:39.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:35:39.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.432781929999997, 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:35:39.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:35:39.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:35:39.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.142 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.405331969000002, 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:39.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:35:39.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:35:39.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:42868 3.1e 3:781b6cfc:::rbd_data.115e56ad4d37.000000000000003c:snapdir [list-snaps] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:39.146 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.360630764, 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021882233000000001, 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 42868 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753558+0000", 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6869999999999999e-06 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753573+0000", 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.446e-05 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753615+0000", 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.227e-05 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753633+0000", 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8240000000000002e-05 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.754227+0000", 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059412099999999997 2024-04-24T00:35:39.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.775439+0000", 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021211455000000001 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72671 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.462350308000001, 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017794547000000001, 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.150 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72671 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:39.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651841+0000", 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651856+0000", 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5549999999999999e-05 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651907+0000", 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1187000000000003e-05 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651933+0000", 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5550000000000001e-05 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652300+0000", 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036690699999999998 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652318+0000", 2024-04-24T00:35:39.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8315999999999999e-05 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669396+0000", 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017078233000000002 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669631+0000", 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000235028 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72679 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.462173195999998, 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019600632999999999, 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72679 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.155 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652016+0000", 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652025+0000", 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8729999999999996e-06 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652047+0000", 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2957000000000001e-05 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652065+0000", 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7289999999999999e-05 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652773+0000", 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070851400000000004 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652791+0000", 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7649999999999999e-05 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671513+0000", 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018722486 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671615+0000", 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010119 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72759 3.a 3:541a2ac3:::rbd_data.115efb339e04.000000000000006f:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.460564398999999, 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017161025999999999, 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.159 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72759 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653624+0000", 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653629+0000", 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1529999999999996e-06 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653660+0000", 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0623999999999997e-05 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653718+0000", 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8514999999999998e-05 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654870+0000", 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001151909 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654881+0000", 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.118e-05 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670564+0000", 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015682413999999999 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670784+0000", 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022022799999999999 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72761 3.a 3:50a1eac1:::rbd_data.115efb339e04.0000000000000072:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:39.163 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.460362058000001, 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017246102999999999, 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72761 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653826+0000", 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9699999999999994e-07 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653843+0000", 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7240000000000001e-05 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653857+0000", 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3823e-05 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653868+0000", 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1055e-05 2024-04-24T00:35:39.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654965+0000", 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001097085 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654987+0000", 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1759999999999998e-05 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670852+0000", 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015864494999999999 2024-04-24T00:35:39.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671071+0000", 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021964799999999999 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72776 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.460284408, 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017442424000000002, 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.168 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72776 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653904+0000", 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1200000000000001e-06 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653914+0000", 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7669999999999998e-06 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653928+0000", 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4165000000000001e-05 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653940+0000", 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-05 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655028+0000", 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010878579999999999 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655033+0000", 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9339999999999997e-06 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671139+0000", 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016105922000000002 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671345+0000", 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020662799999999999 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.172 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72783 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.460230834000001, 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017776166999999999, 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.173 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72783 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653957+0000", 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1699999999999997e-07 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653962+0000", 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6079999999999998e-06 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653975+0000", 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2974e-05 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653986+0000", 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0851e-05 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654503+0000", 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051748099999999997 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654523+0000", 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0032999999999999e-05 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671644+0000", 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017121032000000001 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671733+0000", 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8270999999999996e-05 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:81511 3.d e111/106 3:b780edbf:::rbd_data.115e6f07caf6.0000000000000006:head v 111'118, mlcod=0'0)", 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:39.177 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.1503356509999998, 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029826869999999998, 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:39.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.65e-07 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966954+0000", 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031022379999999998 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966976+0000", 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1565000000000001e-05 2024-04-24T00:35:39.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966988+0000", 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2428e-05 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967105+0000", 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011725799999999999 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967132+0000", 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6863999999999999e-05 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993615+0000", 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026482946 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993678+0000", 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3206000000000004e-05 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84130 3.11 e111/106 3:8ac0652c:::rbd_data.115e6f07caf6.000000000000000b:head v 111'344, mlcod=108'342)", 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.9437926240000003, 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037411427999999997, 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0699999999999999e-07 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171887+0000", 2024-04-24T00:35:39.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00149255 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171909+0000", 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1270000000000001e-05 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171918+0000", 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6139999999999998e-06 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172147+0000", 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022887100000000001 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172341+0000", 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019398700000000001 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207639+0000", 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035298445999999997 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207806+0000", 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016648300000000001 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84132 3.14 e111/106 3:2c6d4ac0:::rbd_data.115e6f07caf6.000000000000000d:head v 111'167, mlcod=0'0)", 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.9339587639999998, 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029744291999999999, 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.186 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180230+0000", 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8029999999999999e-06 2024-04-24T00:35:39.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.184974+0000", 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047440540000000002 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185003+0000", 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8860000000000002e-05 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185021+0000", 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7473999999999999e-05 2024-04-24T00:35:39.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185169+0000", 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148305 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185196+0000", 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7444e-05 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209925+0000", 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024728208000000002 2024-04-24T00:35:39.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209973+0000", 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8143999999999998e-05 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.485260137, 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.190 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:35:39.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:35:39.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.4789888790000001, 2024-04-24T00:35:39.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.199 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.4675342929999999, 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:39.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:35:39.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:35:39.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:35:39.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.4130817490000001, 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:39.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:35:39.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:35:39.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:35:39.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.208 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:39.208 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:39.208 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:39.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:39.209 INFO:teuthology.orchestra.run.smithi064.stdout:3252 writesame 0xab45b87 thru 0xab50f01 data_size 0x1(0xb37b bytes) 2024-04-24T00:35:39.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:39.210 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:39.203+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:39.310 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:39.305+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:39.316 INFO:teuthology.orchestra.run.smithi064.stdout:3253 read 0xbcbdf13 thru 0xbcbf0f8 (0x11e6 bytes) 2024-04-24T00:35:39.317 INFO:teuthology.orchestra.run.smithi064.stdout:3254 read 0x127d73d thru 0x12883a1 (0xac65 bytes) 2024-04-24T00:35:39.319 INFO:teuthology.orchestra.run.smithi064.stdout:3255 write 0xb82c213 thru 0xb831827 (0x5615 bytes) 2024-04-24T00:35:39.327 INFO:teuthology.orchestra.run.smithi064.stdout:3256 write 0x10ec706 thru 0x10f5a92 (0x938d bytes) 2024-04-24T00:35:39.335 INFO:teuthology.orchestra.run.smithi064.stdout:3257 trunc from 0xc2c3e49 to 0xb80608b 2024-04-24T00:35:39.349 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.349 INFO:teuthology.orchestra.run.smithi064.stdout:3260 write 0x717e60c thru 0x718a1e9 (0xbbde bytes) 2024-04-24T00:35:39.363 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:39.363 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:39.363 INFO:teuthology.orchestra.run.smithi064.stdout:3261 trunc from 0xb80608b to 0x6e666dc 2024-04-24T00:35:39.381 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:39.382 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:39.386 INFO:teuthology.orchestra.run.smithi064.stdout:3262 punch from 0x3959a5d to 0x395ec82, (0x5225 bytes) 2024-04-24T00:35:39.386 INFO:teuthology.orchestra.run.smithi064.stdout:3263 read 0x241e942 thru 0x2427475 (0x8b34 bytes) 2024-04-24T00:35:39.388 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:39.388 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:39.388 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:39.388 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:39.396 INFO:teuthology.orchestra.run.smithi064.stdout:3264 write 0x74db98d thru 0x74e7fe7 (0xc65b bytes) 2024-04-24T00:35:39.398 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:39.399 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:39.399 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:39.416 INFO:teuthology.orchestra.run.smithi064.stdout:3266 write 0xd165bae thru 0xd169b46 (0x3f99 bytes) 2024-04-24T00:35:39.417 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:39.418 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:39.420 INFO:teuthology.orchestra.run.smithi064.stdout:3267 write 0xabcd13d thru 0xabcdd9d (0xc61 bytes) 2024-04-24T00:35:39.424 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.424 INFO:teuthology.orchestra.run.smithi064.stdout:3270 punch from 0xc508c1 to 0xc54842, (0x3f81 bytes) 2024-04-24T00:35:39.424 INFO:teuthology.orchestra.run.smithi064.stdout:3271 punch from 0x6a818a3 to 0x6a887da, (0x6f37 bytes) 2024-04-24T00:35:39.424 INFO:teuthology.orchestra.run.smithi064.stdout:3272 punch from 0x245a0cc to 0x2460534, (0x6468 bytes) 2024-04-24T00:35:39.432 INFO:teuthology.orchestra.run.smithi064.stdout:3274 trunc from 0xd169b47 to 0x5cee42e 2024-04-24T00:35:39.437 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:39.437 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:39.450 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:39.447+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:39.460 INFO:teuthology.orchestra.run.smithi064.stdout:3275 trunc from 0x5cee42e to 0xc69073e 2024-04-24T00:35:39.460 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:39.460 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:39.467 INFO:teuthology.orchestra.run.smithi064.stdout:3276 punch from 0x2e6072a to 0x2e6d8a8, (0xd17e bytes) 2024-04-24T00:35:39.467 INFO:teuthology.orchestra.run.smithi064.stdout:3277 read 0x35f156c thru 0x36009f1 (0xf486 bytes) 2024-04-24T00:35:39.467 INFO:teuthology.orchestra.run.smithi064.stdout:3278 read 0x238c59c thru 0x2394c48 (0x86ad bytes) 2024-04-24T00:35:39.468 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.468 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.468 INFO:teuthology.orchestra.run.smithi064.stdout:3282 writesame 0xc0554a5 thru 0xc057884 data_size 0x8(0x23e0 bytes) 2024-04-24T00:35:39.472 INFO:teuthology.orchestra.run.smithi064.stdout:3284 write 0x72b0e6a thru 0x72bd325 (0xc4bc bytes) 2024-04-24T00:35:39.477 INFO:teuthology.orchestra.run.smithi064.stdout:3285 read 0x838965e thru 0x838edc7 (0x576a bytes) 2024-04-24T00:35:39.478 INFO:teuthology.orchestra.run.smithi064.stdout:3286 write 0x95e9630 thru 0x95f6ba5 (0xd576 bytes) 2024-04-24T00:35:39.482 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:39.482 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:39.500 INFO:teuthology.orchestra.run.smithi064.stdout:3289 trunc from 0xc69073e to 0x35fcff3 2024-04-24T00:35:39.504 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:39.504 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:39.511 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:39.507+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:39.521 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.521 INFO:teuthology.orchestra.run.smithi064.stdout:3293 read 0x2850400 thru 0x28514eb (0x10ec bytes) 2024-04-24T00:35:39.527 INFO:teuthology.orchestra.run.smithi064.stdout:3295 trunc from 0x35fcff3 to 0x6ee6506 2024-04-24T00:35:39.537 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:39.537 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:39.537 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.538 INFO:teuthology.orchestra.run.smithi064.stdout:3297 trunc from 0x6ee6506 to 0x74b0239 2024-04-24T00:35:39.547 INFO:teuthology.orchestra.run.smithi064.stdout:3298 read 0x2c91655 thru 0x2c92ba2 (0x154e bytes) 2024-04-24T00:35:39.548 INFO:teuthology.orchestra.run.smithi064.stdout:3299 write 0x6d1d5a4 thru 0x6d25a23 (0x8480 bytes) 2024-04-24T00:35:39.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:39.553 INFO:teuthology.orchestra.run.smithi064.stdout:3301 punch from 0x235e9c0 to 0x2362a08, (0x4048 bytes) 2024-04-24T00:35:39.554 INFO:teuthology.orchestra.run.smithi064.stdout:3303 read 0x61d0ed2 thru 0x61dc0bf (0xb1ee bytes) 2024-04-24T00:35:39.554 INFO:teuthology.orchestra.run.smithi064.stdout:3304 read 0xd8c0d1 thru 0xd92ccb (0x6bfb bytes) 2024-04-24T00:35:39.555 INFO:teuthology.orchestra.run.smithi064.stdout:3305 read 0xaece9e thru 0xaf340b (0x656e bytes) 2024-04-24T00:35:39.556 INFO:teuthology.orchestra.run.smithi064.stdout:3306 write 0x5973b53 thru 0x597b0c1 (0x756f bytes) 2024-04-24T00:35:39.559 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.582 INFO:teuthology.orchestra.run.smithi064.stdout:3308 writesame 0xd3ebcd8 thru 0xd3f1f55 data_size 0x7(0x627e bytes) 2024-04-24T00:35:39.582 INFO:teuthology.orchestra.run.smithi064.stdout:3309 write 0x545c468 thru 0x5463234 (0x6dcd bytes) 2024-04-24T00:35:39.585 INFO:teuthology.orchestra.run.smithi064.stdout:3311 writesame 0xa545266 thru 0xa5481a2 data_size 0x1d(0x2f3d bytes) 2024-04-24T00:35:39.592 INFO:teuthology.orchestra.run.smithi064.stdout:3313 trunc from 0xd3f1f56 to 0xcb599bb 2024-04-24T00:35:39.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:35:39.601 INFO:teuthology.orchestra.run.smithi064.stdout:3314 read 0x1a402d2 thru 0x1a4c9f3 (0xc722 bytes) 2024-04-24T00:35:39.602 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:39.602 INFO:teuthology.orchestra.run.smithi064.stdout:3317 read 0x1d2934c thru 0x1d315ab (0x8260 bytes) 2024-04-24T00:35:39.615 INFO:teuthology.orchestra.run.smithi064.stdout:3318 write 0xe4752d3 thru 0xe47572c (0x45a bytes) 2024-04-24T00:35:39.616 INFO:teuthology.orchestra.run.smithi064.stdout:3319 write 0xe3db05 thru 0xe47733 (0x9c2f bytes) 2024-04-24T00:35:39.620 INFO:teuthology.orchestra.run.smithi064.stdout:3320 punch from 0x44a0218 to 0x44a453f, (0x4327 bytes) 2024-04-24T00:35:39.620 INFO:teuthology.orchestra.run.smithi064.stdout:3322 clone 22 order 21 su 262144 sc 10 2024-04-24T00:35:39.638 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:39.635+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:39.714 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:39.711+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:39.716 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:39.716 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:39.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:39.815 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:39.811+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:39.847 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:39.847 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:39.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:39.916 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:39.911+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:39.970 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:39.971 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:39.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:40.029 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:40.025+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:40.102 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:40.103 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:40.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:40.119 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:40.115+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:40.219 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:40.215+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:40.289 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:40.289 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:40.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:40.350 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:40.345+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:40.420 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:40.415+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:40.427 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:40.428 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:40.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:40.525 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:40.521+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:40.582 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:35:40.613 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:40.613 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:40.622 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:40.619+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:40.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:40.723 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:40.717+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:40.795 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:40.795 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:40.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:40.854 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:40.849+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:40.925 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:40.917+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:40.979 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:40.979 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:40.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:41.038 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:41.033+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:41.126 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:41.123+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:41.158 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:41.158 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:41.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:41.227 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:41.221+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:41.292 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:41.292 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:41.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:41.354 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:41.349+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:41.429 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:41.423+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:41.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:41.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:41.529 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:41.523+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:41.620 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:41.620 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:41.630 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:41.625+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:41.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:41.712 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:41.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:41.731 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:41.727+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:41.733 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:41.733 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:41.757 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:41.757 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:41.776 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:41.777 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:41.807 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:41.808 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:41.824 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:41.824 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:88807 3.6 e118/59 3:616c8299:::rbd_object_map.115e6713a289.000000000000002f:head v 118'253, mlcod=118'250)", 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:41.823203+0000", 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00036349499999999997, 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038813399999999999, 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:41.823203+0000", 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:41.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:41.823203+0000", 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:41.823204+0000", 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2600000000000002e-07 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:41.823208+0000", 2024-04-24T00:35:41.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1230000000000002e-06 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:41.823216+0000", 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4330000000000004e-06 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:41.823222+0000", 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3469999999999998e-06 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:41.823268+0000", 2024-04-24T00:35:41.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5847000000000003e-05 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:41.823290+0000", 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2563000000000001e-05 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:35:41.828 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:41.832 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:41.827+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:41.834 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:41.835 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:41.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:41.856 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:41.856 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:41.874 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:41.874 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:41.934 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:41.931+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:42.000 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:42.000 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:42.000 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:42.000 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:42.000 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:42.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:42.058 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:42.055+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:42.135 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:42.131+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 88.962725171000002, 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:35:42.182 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:35:42.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:10260 3.9 e40/21 ondisk, result = 0)", 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 88.947195922000006, 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061903349000000003, 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229854+0000", 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229856+0000", 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229863+0000", 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7599999999999997e-06 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229890+0000", 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7852999999999999e-05 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.229904+0000", 2024-04-24T00:35:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3833e-05 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291734+0000", 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061830095000000002 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291744+0000", 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2939999999999994e-06 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291758+0000", 2024-04-24T00:35:42.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3841e-05 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10591 3.1c 3:3f6f08fb:::rbd_data.115e380d621c.000000000000011b:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 88.318472602, 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054180482000000002, 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10591 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858577+0000", 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858580+0000", 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.74e-06 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858595+0000", 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4725999999999999e-05 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858629+0000", 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483000000000002e-05 2024-04-24T00:35:42.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858651+0000", 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1667000000000001e-05 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858799+0000", 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014807899999999999 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859162+0000", 2024-04-24T00:35:42.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000362343 2024-04-24T00:35:42.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:42.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.863190+0000", 2024-04-24T00:35:42.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040283070000000001 2024-04-24T00:35:42.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912398+0000", 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049208033999999998 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912502+0000", 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104145 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912643+0000", 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140526 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.912758+0000", 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115432 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:42.198 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.540939476999995, 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:42.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:35:42.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:35:42.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:35:42.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:35:42.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:35:42.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.527116519000003, 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.205 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:35:42.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:35:42.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:35:42.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.299975333999996, 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:22567 3.14 e63/21 ondisk, result = 0)", 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.288185683999998, 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054826831999999999, 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888864+0000", 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888865+0000", 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9099999999999991e-07 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888870+0000", 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7249999999999997e-06 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888882+0000", 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2172e-05 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.888890+0000", 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2960000000000001e-06 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943578+0000", 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054688015 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943585+0000", 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0450000000000003e-06 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943691+0000", 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106588 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33518 3.b 3:d58e132b:::rbd_data.115ec4f9b3a7.000000000000000c:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.294145620000002, 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053502986000000002, 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33518 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882904+0000", 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882906+0000", 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882914+0000", 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9100000000000005e-06 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882936+0000", 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1926999999999999e-05 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.882947+0000", 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1250000000000001e-05 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883009+0000", 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1629000000000002e-05 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.883065+0000", 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5954e-05 2024-04-24T00:35:42.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.885276+0000", 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022118620000000002 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936204+0000", 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050927713999999999 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936308+0000", 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010385799999999999 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936362+0000", 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4339000000000003e-05 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.936407+0000", 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5157999999999997e-05 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.398811910999999, 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.225 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:35:42.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:35:42.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:42.231 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2610401289999995, 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:35:42.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2235578109999992, 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.237 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:35:42.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:35:42.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:35:42.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:81510 3.6 e111/59 3:6416f24e:::rbd_data.115e6f07caf6.0000000000000004:head v 111'174, mlcod=108'172)", 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.975662+0000", 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2013883950000004, 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051816996999999997, 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.975662+0000", 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.975662+0000", 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.975664+0000", 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0150000000000002e-06 2024-04-24T00:35:42.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.981634+0000", 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005970365 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.981659+0000", 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4564999999999999e-05 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.981672+0000", 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3266e-05 2024-04-24T00:35:42.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.994361+0000", 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012688708999999999 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.994385+0000", 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4603000000000001e-05 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027449+0000", 2024-04-24T00:35:42.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033063412 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027479+0000", 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0062000000000001e-05 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.1942288489999999, 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:35:42.247 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:35:42.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:35:42.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84125 3.9 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.151724+0000", 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.025326068, 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064059736000000006, 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84125 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.151724+0000", 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.151724+0000", 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.151728+0000", 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8099999999999999e-06 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.155992+0000", 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004264164 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156017+0000", 2024-04-24T00:35:42.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5103e-05 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156033+0000", 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6342e-05 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156157+0000", 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012311900000000001 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156281+0000", 2024-04-24T00:35:42.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012401600000000001 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.156381+0000", 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100079 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.175498+0000", 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019117173000000001 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.215721+0000", 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040223354000000003 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.215756+0000", 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4483999999999997e-05 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.215784+0000", 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8092000000000001e-05 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84130 3.11 3:8ac0652c:::rbd_data.115e6f07caf6.000000000000000b:head [write 0~4030464 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.165649+0000", 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.011400879, 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053811444999999999, 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.259 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84130 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.165649+0000", 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.165649+0000", 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.165651+0000", 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1509999999999998e-06 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.169750+0000", 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040984439999999997 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.169774+0000", 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4247e-05 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.169786+0000", 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2354000000000001e-05 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.169868+0000", 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2076999999999994e-05 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170049+0000", 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018052699999999999 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170219+0000", 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017027700000000001 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.215809+0000", 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045589835000000002 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219354+0000", 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035447389999999999 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219412+0000", 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7655999999999998e-05 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219461+0000", 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9138000000000003e-05 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:42.264 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.9969669459999997, 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:35:42.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5612119470000003, 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:35:42.270 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:35:42.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:35:42.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:35:42.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5513539109999996, 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.276 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:35:42.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:35:42.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5310043029999996, 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:35:42.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:35:42.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87736 3.a e115/21 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head v 115'512, mlcod=111'510)", 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.652456+0000", 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5245936169999998, 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061811680000000001, 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652456+0000", 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652456+0000", 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652459+0000", 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.656878+0000", 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044193660000000001 2024-04-24T00:35:42.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.656908+0000", 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9828e-05 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.656925+0000", 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7674000000000001e-05 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.657043+0000", 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117351 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.657074+0000", 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1157e-05 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.714206+0000", 2024-04-24T00:35:42.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057132170000000003 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.714268+0000", 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2033999999999996e-05 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.295 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:42.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:42.297 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:42.289+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:42.397 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:42.391+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:42.464 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:42.464 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:42.464 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:42.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:42.526 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:42.523+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:42.599 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:42.597+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:42.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:35:42.657 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:42.658 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:42.658 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:42.658 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:42.658 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:42.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:42.699 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:42.693+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:42.795 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-04-24T00:35:42.797 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:42.797 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:42.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:42.797 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:42.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.797 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:42.797 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.085753526000005, 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:35:42.802 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.085655600999999, 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.803 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:35:42.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:35:42.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.726276069000001, 2024-04-24T00:35:42.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:35:42.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:35:42.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:35:42.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30133 3.1f e66/59 ondisk, result = 0)", 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.713682521999999, 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060466079999999998, 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.815 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080440+0000", 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:35:42.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080446+0000", 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6699999999999999e-06 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080465+0000", 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080476+0000", 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0937999999999999e-05 2024-04-24T00:35:42.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140869+0000", 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060392759999999997 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140882+0000", 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3131e-05 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140905+0000", 2024-04-24T00:35:42.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2642000000000001e-05 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30705 3.2 3:45c09e0c:::rbd_data.115e8210e51a.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.541669607999999, 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063528001000000001, 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.819 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30705 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252455+0000", 2024-04-24T00:35:42.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.303e-06 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252477+0000", 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1970000000000001e-05 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252515+0000", 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7616e-05 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252536+0000", 2024-04-24T00:35:42.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376999999999999e-05 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252696+0000", 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160192 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252945+0000", 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024906400000000001 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.256753+0000", 2024-04-24T00:35:42.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038072129999999998 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315658+0000", 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058905258000000002 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315790+0000", 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013223999999999999 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315894+0000", 2024-04-24T00:35:42.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103886 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315980+0000", 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5882000000000002e-05 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30705 3.2 e66/53 ondisk, result = 0)", 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:42.824 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.526457505000003, 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048330598000000002, 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:42.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267667+0000", 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.96e-06 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267683+0000", 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6336000000000001e-05 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267730+0000", 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7079000000000001e-05 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267757+0000", 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6364e-05 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315724+0000", 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047967018 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315742+0000", 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7935999999999999e-05 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315995+0000", 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025290499999999999 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33506 3.c 3:3787ff34:::rbd_data.115ec4f9b3a7.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.996073463999998, 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061037313000000003, 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33506 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798052+0000", 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:35:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798067+0000", 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5160000000000001e-05 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798107+0000", 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9946e-05 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798130+0000", 2024-04-24T00:35:42.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2637e-05 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798238+0000", 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010841299999999999 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798344+0000", 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010631199999999999 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:42.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.801407+0000", 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030623600000000001 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858616+0000", 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057209342000000003 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858879+0000", 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000262952 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858992+0000", 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113429 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859085+0000", 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2925999999999998e-05 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33506 3.c e70/56 ondisk, result = 0)", 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:42.834 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.986440696999999, 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051419185999999999, 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:42.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807682+0000", 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807687+0000", 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807704+0000", 2024-04-24T00:35:42.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7424000000000001e-05 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807714+0000", 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3400000000000004e-06 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858714+0000", 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051000248999999998 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858732+0000", 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7995999999999998e-05 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859100+0000", 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036787400000000002 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35694 3.1b 3:dcf0eb61:::rbd_data.115ec5833ea9.00000000000000aa:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:42.838 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.874624846000003, 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049399446, 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35694 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919498+0000", 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919504+0000", 2024-04-24T00:35:42.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919520+0000", 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5908000000000001e-05 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919532+0000", 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2964e-05 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919630+0000", 2024-04-24T00:35:42.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7096999999999995e-05 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919734+0000", 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104128 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.922056+0000", 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023219149999999999 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968775+0000", 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046719274999999998 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968817+0000", 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-05 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968858+0000", 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1440999999999998e-05 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968896+0000", 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7741000000000003e-05 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:42.844 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 45.903292082, 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.845 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:35:42.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44664 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 524288~131072 [fadvise_sequential+fadvise_nocache],write 1179648~1048576 [fadvise_sequential+fadvise_nocache],write 10616832~2048 [fadvise_sequential+fadvise_nocache],write 10627072~4096 [fadvise_sequential+fadvise_nocache],write 10639360~4096 [fadvise_sequential+fadvise_nocache],write 10651648~4096 [fadvise_sequential+fadvise_nocache],write 10663936~4096 [fadvise_sequential+fadvise_nocache],write 10676224~4096 [fadvise_sequential+fadvise_nocache],write 10688512~4096 [fadvise_sequential+fadvise_nocache],write 10700800~4096 [fadvise_sequential+fadvise_nocache],write 10713088~4096 [fadvise_sequential+fadvise_nocache],write 10725376~4096 [fadvise_sequential+fadvise_nocache],write 10737664~4096 [fadvise_sequential+fadvise_nocache],write 10754048~4096 [fadvise_sequential+fadvise_nocache],write 10766336~4096 [fadvise_sequential+fadvise_nocache],write 10778624~4096 [fadvise_sequential+fadvise_nocache],write 10790912~4096 [fadvise_sequential+fadvise_nocache],write 10803200~4096 [fadvise_sequential+fadvise_nocache],write 10815488~4096 [fadvise_sequential+fadvise_nocache],write 10827776~4096 [fadvise_sequential+fadvise_nocache],write 10840064~4096 [fadvise_sequential+fadvise_nocache],write 10852352~4096 [fadvise_sequential+fadvise_nocache],write 10864640~4096 [fadvise_sequential+fadvise_nocache],write 10876928~2048 [fadvise_sequential+fadvise_nocache],write 10881024~4096 [fadvise_sequential+fadvise_nocache],write 10893312~4096 [fadvise_sequential+fadvise_nocache],write 10905600~4096 [fadvise_sequential+fadvise_nocache],write 10917888~4096 [fadvise_sequential+fadvise_nocache],write 10930176~4096 [fadvise_sequential+fadvise_nocache],write 10942464~4096 [fadvise_sequential+fadvise_nocache],write 10954752~4096 [fadvise_sequential+fadvise_nocache],write 10967040~4096 [fadvise_sequential+fadvise_nocache],write 10979328~4096 [fadvise_sequential+fadvise_nocache],write 10991616~4096 [fadvise_sequential+fadvise_nocache],write 11003904~4096 [fadvise_sequential+fadvise_nocache],write 11010048~2048 [fadvise_sequential+fadvise_nocache],write 11020288~4096 [fadvise_sequential+fadvise_nocache],write 11032576~4096 [fadvise_sequential+fadvise_nocache],write 11044864~4096 [fadvise_sequential+fadvise_nocache],write 11057152~4096 [fadvise_sequential+fadvise_nocache],write 11069440~4096 [fadvise_sequential+fadvise_nocache],write 11081728~4096 [fadvise_sequential+fadvise_nocache],write 11094016~4096 [fadvise_sequential+fadvise_nocache],write 11106304~4096 [fadvise_sequential+fadvise_nocache],write 11118592~4096 [fadvise_sequential+fadvise_nocache],write 11130880~4096 [fadvise_sequential+fadvise_nocache],write 14417920~4096 [fadvise_sequential+fadvise_nocache],write 14426112~8192 [fadvise_sequential+fadvise_nocache],write 14438400~8192 [fadvise_sequential+fadvise_nocache],write 14450688~8192 [fadvise_sequential+fadvise_nocache],write 14462976~8192 [fadvise_sequential+fadvise_nocache],write 14475264~8192 [fadvise_sequential+fadvise_nocache],write 14487552~8192 [fadvise_sequential+fadvise_nocache],write 14499840~8192 [fadvise_sequential+fadvise_nocache],write 14512128~8192 [fadvise_sequential+fadvise_nocache],write 14524416~8192 [fadvise_sequential+fadvise_nocache],write 14536704~8192 [fadvise_sequential+fadvise_nocache],write 14553088~8192 [fadvise_sequential+fadvise_nocache],write 14565376~8192 [fadvise_sequential+fadvise_nocache],write 14577664~8192 [fadvise_sequential+fadvise_nocache],write 14589952~8192 [fadvise_sequential+fadvise_nocache],write 14602240~8192 [fadvise_sequential+fadvise_nocache],write 14614528~8192 [fadvise_sequential+fadvise_nocache],write 14626816~8192 [fadvise_sequential+fadvise_nocache],write 14639104~8192 [fadvise_sequential+fadvise_nocache],write 14651392~8192 [fadvise_sequential+fadvise_nocache],write 14663680~8192 [fadvise_sequential+fadvise_nocache],write 14675968~4096 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:42.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 45.891218760999998, 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768734, 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44664 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:42.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902904+0000", 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5179999999999999e-06 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.904804+0000", 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018996390000000001 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905393+0000", 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058872600000000003 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905408+0000", 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5073e-05 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905482+0000", 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4284999999999998e-05 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905785+0000", 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030266100000000002 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.907249+0000", 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001464899 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.932396+0000", 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025146379 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958308+0000", 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025912418999999999 2024-04-24T00:35:42.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958527+0000", 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218297 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958671+0000", 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144838 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.315531943000003, 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:35:42.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:35:42.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:35:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.290263336999999, 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:35:42.862 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:67305 3.17 e89/59 ondisk, result = 0)", 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.939489260999999, 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049974565999999998, 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.866 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854633+0000", 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854637+0000", 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.946e-06 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854649+0000", 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2507000000000001e-05 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854658+0000", 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6330000000000002e-06 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904468+0000", 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049809535000000002 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904479+0000", 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-05 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904607+0000", 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012738300000000001 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.437059616999999, 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.870 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:35:42.875 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.404454503, 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:35:42.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.999879808999999, 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.942039578999999, 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:42.885 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:35:42.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:35:42.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81512 3.5 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.8309064450000001, 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061553231999999999, 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.889 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81512 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963217+0000", 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9439999999999999e-06 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967964+0000", 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047468529999999997 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967986+0000", 2024-04-24T00:35:42.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1852e-05 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968000+0000", 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4426e-05 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968069+0000", 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9338000000000006e-05 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968219+0000", 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014914999999999999 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968363+0000", 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144093 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993999+0000", 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025636744999999999 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024681+0000", 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030681452000000001 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024722+0000", 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1115999999999997e-05 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024768+0000", 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6263000000000001e-05 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.894 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84133 3.1f 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head [write 0~4030464 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.6216560930000004, 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060722168, 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:42.895 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84133 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:42.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172468+0000", 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2869999999999999e-06 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180251+0000", 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0077834189999999998 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180276+0000", 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4932000000000001e-05 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180288+0000", 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2224e-05 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180378+0000", 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9401999999999995e-05 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180542+0000", 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016484599999999999 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180700+0000", 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015754300000000001 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.208480+0000", 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027780143 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233097+0000", 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02461675 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233151+0000", 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4593999999999999e-05 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233187+0000", 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6028000000000001e-05 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:42.900 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:42.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:42.902 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:42.897+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:43.003 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:42.997+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:43.026 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:43.027 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:43.027 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:43.027 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:43.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:43.041 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:35:43.041 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:35:43.041 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.2503296576428551 2024-04-24T00:35:43.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 0.2503296576428551 2024-04-24T00:35:43.103 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:43.099+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:43.162 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:43.162 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:43.162 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:43.162 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:43.162 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:43.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:43.204 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:43.197+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:43.247 INFO:teuthology.orchestra.run.smithi064.stdout:3323 read 0x7b6f627 thru 0x7b7278e (0x3168 bytes) 2024-04-24T00:35:43.251 INFO:teuthology.orchestra.run.smithi064.stdout:3324 trunc from 0xe47572d to 0xc21d53b 2024-04-24T00:35:43.291 INFO:teuthology.orchestra.run.smithi064.stdout:3326 punch from 0x7c8ad58 to 0x7c90972, (0x5c1a bytes) 2024-04-24T00:35:43.291 INFO:teuthology.orchestra.run.smithi064.stdout:3327 punch from 0x2a65bcb to 0x2a698f0, (0x3d25 bytes) 2024-04-24T00:35:43.291 INFO:teuthology.orchestra.run.smithi064.stdout:3328 read 0x7470fff thru 0x747464c (0x364e bytes) 2024-04-24T00:35:43.291 INFO:teuthology.orchestra.run.smithi064.stdout:3329 punch from 0x6629935 to 0x6633112, (0x97dd bytes) 2024-04-24T00:35:43.292 INFO:teuthology.orchestra.run.smithi064.stdout:3330 write 0x7d4204e thru 0x7d4f3b4 (0xd367 bytes) 2024-04-24T00:35:43.303 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:43.306 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:43.299+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:43.314 INFO:teuthology.orchestra.run.smithi064.stdout:3332 trunc from 0xc21d53b to 0x4a65e86 2024-04-24T00:35:43.356 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:43.356 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:43.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:43.356 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.819811436999998, 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.357 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:35:43.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.362 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.785774384000007, 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:35:43.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 62.195048419999999, 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:43.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:35:43.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 62.185792294999999, 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:43.373 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:35:43.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:43.377 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.670660388000002, 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.643210427, 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.383 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:35:43.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.387 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:42868 3.1e 3:781b6cfc:::rbd_data.115e56ad4d37.000000000000003c:snapdir [list-snaps] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.598509221999997, 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021882233000000001, 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.388 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 42868 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753558+0000", 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6869999999999999e-06 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753573+0000", 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.446e-05 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753615+0000", 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.227e-05 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753633+0000", 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8240000000000002e-05 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.754227+0000", 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059412099999999997 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.775439+0000", 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021211455000000001 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72671 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.700228765999999, 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017794547000000001, 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.392 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72671 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651841+0000", 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651856+0000", 2024-04-24T00:35:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5549999999999999e-05 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651907+0000", 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1187000000000003e-05 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651933+0000", 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5550000000000001e-05 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652300+0000", 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036690699999999998 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652318+0000", 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8315999999999999e-05 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669396+0000", 2024-04-24T00:35:43.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017078233000000002 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669631+0000", 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000235028 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72679 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.700051653999999, 2024-04-24T00:35:43.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019600632999999999, 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72679 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652016+0000", 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652025+0000", 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8729999999999996e-06 2024-04-24T00:35:43.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652047+0000", 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2957000000000001e-05 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652065+0000", 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7289999999999999e-05 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652773+0000", 2024-04-24T00:35:43.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070851400000000004 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652791+0000", 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7649999999999999e-05 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671513+0000", 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018722486 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671615+0000", 2024-04-24T00:35:43.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010119 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72759 3.a 3:541a2ac3:::rbd_data.115efb339e04.000000000000006f:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.698442857, 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017161025999999999, 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.402 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72759 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653624+0000", 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653629+0000", 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1529999999999996e-06 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653660+0000", 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0623999999999997e-05 2024-04-24T00:35:43.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653718+0000", 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8514999999999998e-05 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654870+0000", 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001151909 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654881+0000", 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.118e-05 2024-04-24T00:35:43.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670564+0000", 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015682413999999999 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670784+0000", 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022022799999999999 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72761 3.a 3:50a1eac1:::rbd_data.115efb339e04.0000000000000072:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.698240515999998, 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017246102999999999, 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72761 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653826+0000", 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9699999999999994e-07 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653843+0000", 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7240000000000001e-05 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653857+0000", 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3823e-05 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653868+0000", 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1055e-05 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654965+0000", 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001097085 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654987+0000", 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1759999999999998e-05 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670852+0000", 2024-04-24T00:35:43.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015864494999999999 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671071+0000", 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021964799999999999 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72776 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.698162866000001, 2024-04-24T00:35:43.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017442424000000002, 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72776 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653904+0000", 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1200000000000001e-06 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653914+0000", 2024-04-24T00:35:43.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7669999999999998e-06 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653928+0000", 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4165000000000001e-05 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653940+0000", 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-05 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655028+0000", 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010878579999999999 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655033+0000", 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9339999999999997e-06 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671139+0000", 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016105922000000002 2024-04-24T00:35:43.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671345+0000", 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020662799999999999 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72783 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.698109292000002, 2024-04-24T00:35:43.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017776166999999999, 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72783 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:43.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653957+0000", 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1699999999999997e-07 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653962+0000", 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6079999999999998e-06 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653975+0000", 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2974e-05 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653986+0000", 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0851e-05 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654503+0000", 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051748099999999997 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654523+0000", 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0032999999999999e-05 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671644+0000", 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017121032000000001 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671733+0000", 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8270999999999996e-05 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:81511 3.d e111/106 3:b780edbf:::rbd_data.115e6f07caf6.0000000000000006:head v 111'118, mlcod=0'0)", 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.3882141089999998, 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029826869999999998, 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.421 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.65e-07 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966954+0000", 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031022379999999998 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966976+0000", 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1565000000000001e-05 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966988+0000", 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2428e-05 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967105+0000", 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011725799999999999 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967132+0000", 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6863999999999999e-05 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993615+0000", 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026482946 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993678+0000", 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3206000000000004e-05 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84130 3.11 e111/106 3:8ac0652c:::rbd_data.115e6f07caf6.000000000000000b:head v 111'344, mlcod=108'342)", 2024-04-24T00:35:43.425 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.1816710819999994, 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037411427999999997, 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0699999999999999e-07 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171887+0000", 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00149255 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171909+0000", 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1270000000000001e-05 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171918+0000", 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6139999999999998e-06 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172147+0000", 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022887100000000001 2024-04-24T00:35:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172341+0000", 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019398700000000001 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207639+0000", 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035298445999999997 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207806+0000", 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016648300000000001 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84132 3.14 e111/106 3:2c6d4ac0:::rbd_data.115e6f07caf6.000000000000000d:head v 111'167, mlcod=0'0)", 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.1718372220000006, 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029744291999999999, 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.430 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180230+0000", 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8029999999999999e-06 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.184974+0000", 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047440540000000002 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185003+0000", 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8860000000000002e-05 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185021+0000", 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7473999999999999e-05 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185169+0000", 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148305 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185196+0000", 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7444e-05 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209925+0000", 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024728208000000002 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209973+0000", 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8143999999999998e-05 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.434 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.723138595, 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:35:43.438 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.716867337, 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.439 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:35:43.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.444 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.7054127509999999, 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:35:43.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.6509602069999998, 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.449 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.453 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:43.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:43.456 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:43.451+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:43.457 INFO:teuthology.orchestra.run.smithi064.stdout:3333 read 0x3719e05 thru 0x37237e4 (0x99e0 bytes) 2024-04-24T00:35:43.457 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:43.457 INFO:teuthology.orchestra.run.smithi064.stdout:3335 writesame 0x2d2be23 thru 0x2d2f3e1 data_size 0x1(0x35bf bytes) 2024-04-24T00:35:43.457 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:43.457 INFO:teuthology.orchestra.run.smithi064.stdout:3337 writesame 0xbffe77e thru 0xc00457b data_size 0x35(0x5dfe bytes) 2024-04-24T00:35:43.457 INFO:teuthology.orchestra.run.smithi064.stdout:3338 read 0x1b6da47 thru 0x1b6de1a (0x3d4 bytes) 2024-04-24T00:35:43.458 INFO:teuthology.orchestra.run.smithi064.stdout:3339 read 0x75a10d1 thru 0x75a8f85 (0x7eb5 bytes) 2024-04-24T00:35:43.458 INFO:teuthology.orchestra.run.smithi064.stdout:3342 writesame 0x8d0fc09 thru 0x8d14aeb data_size 0x7(0x4ee3 bytes) 2024-04-24T00:35:43.458 INFO:teuthology.orchestra.run.smithi064.stdout:3343 punch from 0x13aae5 to 0x13fda5, (0x52c0 bytes) 2024-04-24T00:35:43.458 INFO:teuthology.orchestra.run.smithi064.stdout:3344 write 0x2733b8f thru 0x273a321 (0x6793 bytes) 2024-04-24T00:35:43.458 INFO:teuthology.orchestra.run.smithi064.stdout:3345 read 0x70bd786 thru 0x70c0a90 (0x330b bytes) 2024-04-24T00:35:43.458 INFO:teuthology.orchestra.run.smithi064.stdout:3347 punch from 0x6e0623d to 0x6e07687, (0x144a bytes) 2024-04-24T00:35:43.458 INFO:teuthology.orchestra.run.smithi064.stdout:3349 read 0xb045c32 thru 0xb0501c0 (0xa58f bytes) 2024-04-24T00:35:43.463 INFO:teuthology.orchestra.run.smithi064.stdout:3350 write 0xe6750cd thru 0xe67dbf7 (0x8b2b bytes) 2024-04-24T00:35:43.469 INFO:teuthology.orchestra.run.smithi064.stdout:3351 read 0x259d8d2 thru 0x259e39c (0xacb bytes) 2024-04-24T00:35:43.470 INFO:teuthology.orchestra.run.smithi064.stdout:3353 writesame 0xe5a0277 thru 0xe5af981 data_size 0x3(0xf70b bytes) 2024-04-24T00:35:43.538 INFO:teuthology.orchestra.run.smithi064.stdout:3354 write 0x30d9f96 thru 0x30e8912 (0xe97d bytes) 2024-04-24T00:35:43.552 INFO:teuthology.orchestra.run.smithi064.stdout:3356 read 0x1c0710f thru 0x1c0b9a9 (0x489b bytes) 2024-04-24T00:35:43.553 INFO:teuthology.orchestra.run.smithi064.stdout:3357 punch from 0xc5d5b43 to 0xc5da3eb, (0x48a8 bytes) 2024-04-24T00:35:43.553 INFO:teuthology.orchestra.run.smithi064.stdout:3359 write 0x18a4d42 thru 0x18ac597 (0x7856 bytes) 2024-04-24T00:35:43.556 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:43.551+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:43.558 INFO:teuthology.orchestra.run.smithi064.stdout:3360 write 0xd75c862 thru 0xd761b8f (0x532e bytes) 2024-04-24T00:35:43.566 INFO:teuthology.orchestra.run.smithi064.stdout:3361 read 0x607084e thru 0x607199b (0x114e bytes) 2024-04-24T00:35:43.567 INFO:teuthology.orchestra.run.smithi064.stdout:3362 writesame 0x2854000 thru 0x285febb data_size 0x4e(0xbebc bytes) 2024-04-24T00:35:43.574 INFO:teuthology.orchestra.run.smithi064.stdout:3363 write 0x3a0c74 thru 0x3a480f (0x3b9c bytes) 2024-04-24T00:35:43.578 INFO:teuthology.orchestra.run.smithi064.stdout:3364 punch from 0xa1b305a to 0xa1ba605, (0x75ab bytes) 2024-04-24T00:35:43.578 INFO:teuthology.orchestra.run.smithi064.stdout:3368 write 0x8c0b1e2 thru 0x8c0dbb6 (0x29d5 bytes) 2024-04-24T00:35:43.585 INFO:teuthology.orchestra.run.smithi064.stdout:3370 punch from 0x8de8c7f to 0x8decb47, (0x3ec8 bytes) 2024-04-24T00:35:43.585 INFO:teuthology.orchestra.run.smithi064.stdout:3371 write 0x37be4cd thru 0x37c288f (0x43c3 bytes) 2024-04-24T00:35:43.593 INFO:teuthology.orchestra.run.smithi064.stdout:3373 write 0xcdbbb4c thru 0xcdbee33 (0x32e8 bytes) 2024-04-24T00:35:43.601 INFO:teuthology.orchestra.run.smithi064.stdout:3374 read 0x23dd6b9 thru 0x23df21f (0x1b67 bytes) 2024-04-24T00:35:43.602 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:43.602 INFO:teuthology.orchestra.run.smithi064.stdout:3377 write 0xad8510d thru 0xad897fc (0x46f0 bytes) 2024-04-24T00:35:43.610 INFO:teuthology.orchestra.run.smithi064.stdout:3379 read 0x81b12ed thru 0x81b2995 (0x16a9 bytes) 2024-04-24T00:35:43.611 INFO:teuthology.orchestra.run.smithi064.stdout:3380 read 0xdb5caaf thru 0xdb618ed (0x4e3f bytes) 2024-04-24T00:35:43.611 INFO:teuthology.orchestra.run.smithi064.stdout:3381 read 0xe2bcac thru 0xe32feb (0x7340 bytes) 2024-04-24T00:35:43.612 INFO:teuthology.orchestra.run.smithi064.stdout:3382 write 0x702e433 thru 0x703c1b6 (0xdd84 bytes) 2024-04-24T00:35:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:35:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 0.25033 (4015) 2024-04-24T00:35:43.636 INFO:teuthology.orchestra.run.smithi064.stdout:3384 read 0x2479063 thru 0x24810cb (0x8069 bytes) 2024-04-24T00:35:43.637 INFO:teuthology.orchestra.run.smithi064.stdout:3385 punch from 0x8e5158c to 0x8e5cfbf, (0xba33 bytes) 2024-04-24T00:35:43.637 INFO:teuthology.orchestra.run.smithi064.stdout:3386 read 0xdcdfe16 thru 0xdce0836 (0xa21 bytes) 2024-04-24T00:35:43.637 INFO:teuthology.orchestra.run.smithi064.stdout:3388 write 0x5841793 thru 0x58456d8 (0x3f46 bytes) 2024-04-24T00:35:43.641 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:43.641 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:43.641 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:43.641 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:43.644 INFO:teuthology.orchestra.run.smithi064.stdout:3389 read 0x5828e39 thru 0x583845a (0xf622 bytes) 2024-04-24T00:35:43.644 INFO:teuthology.orchestra.run.smithi064.stdout:3390 writesame 0xc454b40 thru 0xc463e39 data_size 0x15(0xf2fa bytes) 2024-04-24T00:35:43.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:43.657 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:43.651+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:43.757 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:43.753+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:43.819 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:43.820 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:43.820 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:43.820 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:43.820 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:43.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:43.878 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:43.875+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:43.893 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:43.953 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.024114732, 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:35:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:35:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:35:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:35:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:35:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:35:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.092149043999996, 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:35:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.090900296000001, 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:35:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:35:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:35:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.933276191999994, 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:35:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:35:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:35:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.919781649000001, 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:35:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.658000533999996, 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:35:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:35:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.404414391000003, 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:35:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:35:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:35:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:35:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.376740526999995, 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:35:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 62.882717706999998, 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:35:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 62.873708872999998, 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.372009925, 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:35:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.371947579, 2024-04-24T00:35:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:35:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:35:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.371437899999997, 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:35:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:35:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:35:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.371326838000002, 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:35:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:35:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.371261572000002, 2024-04-24T00:35:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:35:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:35:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.371201505999998, 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.371144813000001, 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:35:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:35:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.371075480000002, 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:35:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.370984860999997, 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:35:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:35:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.370941448000004, 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:35:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:44.057 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:44.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:44.059 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:44.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:44.060 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:44.055+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:44.160 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:44.153+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:44.179 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:44.179 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:44.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:44.285 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:44.281+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:44.290 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:44.290 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:44.313 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:44.313 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:44.337 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:44.338 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:44.362 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:44.357+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:44.363 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:44.363 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:44.373 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:44.374 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:44.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:44.389 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:44.414 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:44.415 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:44.442 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:44.442 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:44.462 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:44.459+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:44.542 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:44.542 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:44.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:44.602 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:44.597+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:44.664 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:44.657+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:44.687 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:44.687 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:44.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:44.789 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:44.785+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:44.866 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:44.863+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:44.873 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:44.874 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:44.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:44.967 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:44.963+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:45.058 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:45.058 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:45.068 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:45.061+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:45.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:45.169 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:45.161+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:45.238 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:45.239 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:45.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:45.270 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:45.267+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:45.371 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:45.365+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:45.425 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:45.425 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:45.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:45.472 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:45.467+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:45.573 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:45.567+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:45.633 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:45.633 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:45.638 INFO:teuthology.orchestra.run.smithi064.stdout:3392 trunc from 0xe67dbf8 to 0xb9052c8 2024-04-24T00:35:45.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:35:45.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:45.663 INFO:teuthology.orchestra.run.smithi064.stdout:3393 write 0xad1c46 thru 0xad8254 (0x660f bytes) 2024-04-24T00:35:45.669 INFO:teuthology.orchestra.run.smithi064.stdout:3395 trunc from 0xb9052c8 to 0xad665c4 2024-04-24T00:35:45.674 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:45.671+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:45.690 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:45.690 INFO:teuthology.orchestra.run.smithi064.stdout:3397 punch from 0x610d214 to 0x61169b0, (0x979c bytes) 2024-04-24T00:35:45.690 INFO:teuthology.orchestra.run.smithi064.stdout:3398 read 0x29562d8 thru 0x2959fbb (0x3ce4 bytes) 2024-04-24T00:35:45.691 INFO:teuthology.orchestra.run.smithi064.stdout:3399 read 0x7d36878 thru 0x7d3a2c5 (0x3a4e bytes) 2024-04-24T00:35:45.691 INFO:teuthology.orchestra.run.smithi064.stdout:3400 read 0x1c870b2 thru 0x1c87151 (0xa0 bytes) 2024-04-24T00:35:45.692 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:45.692 INFO:teuthology.orchestra.run.smithi064.stdout:3404 read 0x113974c thru 0x113d03a (0x38ef bytes) 2024-04-24T00:35:45.693 INFO:teuthology.orchestra.run.smithi064.stdout:3405 read 0x260b07d thru 0x261511d (0xa0a1 bytes) 2024-04-24T00:35:45.709 INFO:teuthology.orchestra.run.smithi064.stdout:3406 writesame 0xe41d470 thru 0xe4248e2 data_size 0x13(0x7473 bytes) 2024-04-24T00:35:45.710 INFO:teuthology.orchestra.run.smithi064.stdout:3407 read 0x9bc338f thru 0x9bccfa0 (0x9c12 bytes) 2024-04-24T00:35:45.710 INFO:teuthology.orchestra.run.smithi064.stdout:3408 read 0x9a5be07 thru 0x9a5c024 (0x21e bytes) 2024-04-24T00:35:45.710 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:45.710 INFO:teuthology.orchestra.run.smithi064.stdout:3413 read 0x398608b thru 0x398f28a (0x9200 bytes) 2024-04-24T00:35:45.710 INFO:teuthology.orchestra.run.smithi064.stdout:3414 write 0x7e25eb8 thru 0x7e2cfd2 (0x711b bytes) 2024-04-24T00:35:45.718 INFO:teuthology.orchestra.run.smithi064.stdout:3415 punch from 0x531c1d9 to 0x532aebb, (0xece2 bytes) 2024-04-24T00:35:45.718 INFO:teuthology.orchestra.run.smithi064.stdout:3416 punch from 0x2201f0d to 0x2204dc5, (0x2eb8 bytes) 2024-04-24T00:35:45.718 INFO:teuthology.orchestra.run.smithi064.stdout:3418 read 0x5edd943 thru 0x5ee3cdf (0x639d bytes) 2024-04-24T00:35:45.718 INFO:teuthology.orchestra.run.smithi064.stdout:3419 write 0x808be86 thru 0x8098aaa (0xcc25 bytes) 2024-04-24T00:35:45.741 INFO:teuthology.orchestra.run.smithi064.stdout:3421 trunc from 0xe4248e3 to 0x41beaf5 2024-04-24T00:35:45.774 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:45.775 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:45.769+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:45.785 INFO:teuthology.orchestra.run.smithi064.stdout:3423 write 0x556ebb2 thru 0x5574026 (0x5475 bytes) 2024-04-24T00:35:45.790 INFO:teuthology.orchestra.run.smithi064.stdout:3424 clone 23 order 24 su 1048576 sc 7 2024-04-24T00:35:45.812 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:45.812 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:45.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:45.875 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:45.869+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:45.976 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:45.971+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:46.013 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:46.013 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:46.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:46.078 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:46.071+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:46.149 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:46.149 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:46.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:46.210 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:46.207+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:46.280 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:46.275+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:46.298 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:46.298 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:46.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:46.402 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:46.399+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:46.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:46.481 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:46.475+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:46.491 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:46.491 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:46.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:46.575 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:35:46.583 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:46.577+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:46.628 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:46.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:46.645 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:46.646 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:46.663 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:46.663 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:46.672 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:46.672 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:46.680 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:46.680 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:46.683 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:46.677+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:46.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:46.698 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:46.698 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:46.718 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:46.718 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:46.736 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:46.737 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:46.755 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:46.755 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:46.784 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:46.779+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:46.810 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:46.810 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:46.810 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:46.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:46.884 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:46.877+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:46.985 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:46.981+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:46.990 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:46.990 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:46.990 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:46.990 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:46.990 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:47.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:47.086 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:47.081+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:47.130 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:47.130 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:47.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:47.131 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:47.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.131 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:47.131 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:47.131 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.418062961000004, 2024-04-24T00:35:47.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:35:47.131 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.132 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.132 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:47.132 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.132 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.132 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.132 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:35:47.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.133 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:47.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:35:47.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:35:47.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:35:47.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:35:47.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:35:47.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:35:47.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:35:47.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:47.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:35:47.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:35:47.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:35:47.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:35:47.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:47.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:35:47.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.417965035999998, 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.141 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:35:47.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:35:47.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.058585504000007, 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:35:47.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:35:47.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30133 3.1f e66/59 ondisk, result = 0)", 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.045991956999998, 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060466079999999998, 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080439+0000", 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080440+0000", 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080446+0000", 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6699999999999999e-06 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080465+0000", 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9536999999999999e-05 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.080476+0000", 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0937999999999999e-05 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140869+0000", 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060392759999999997 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140882+0000", 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3131e-05 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140905+0000", 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2642000000000001e-05 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30705 3.2 3:45c09e0c:::rbd_data.115e8210e51a.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:47.156 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.873979043000006, 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063528001000000001, 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30705 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252452+0000", 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252455+0000", 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.303e-06 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252477+0000", 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1970000000000001e-05 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252515+0000", 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7616e-05 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252536+0000", 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376999999999999e-05 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252696+0000", 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160192 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.252945+0000", 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024906400000000001 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.256753+0000", 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038072129999999998 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315658+0000", 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058905258000000002 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315790+0000", 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013223999999999999 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315894+0000", 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103886 2024-04-24T00:35:47.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315980+0000", 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5882000000000002e-05 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30705 3.2 e66/53 ondisk, result = 0)", 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.858766939999995, 2024-04-24T00:35:47.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048330598000000002, 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267664+0000", 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267667+0000", 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.96e-06 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267683+0000", 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6336000000000001e-05 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267730+0000", 2024-04-24T00:35:47.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7079000000000001e-05 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.267757+0000", 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6364e-05 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315724+0000", 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047967018 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315742+0000", 2024-04-24T00:35:47.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7935999999999999e-05 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:42.315995+0000", 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025290499999999999 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:33506 3.c 3:3787ff34:::rbd_data.115ec4f9b3a7.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:47.166 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.328382898999998, 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061037313000000003, 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 33506 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798048+0000", 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798052+0000", 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798067+0000", 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5160000000000001e-05 2024-04-24T00:35:47.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798107+0000", 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9946e-05 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798130+0000", 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2637e-05 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798238+0000", 2024-04-24T00:35:47.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010841299999999999 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.798344+0000", 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010631199999999999 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.801407+0000", 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030623600000000001 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858616+0000", 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057209342000000003 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858879+0000", 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000262952 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858992+0000", 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113429 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859085+0000", 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2925999999999998e-05 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:33506 3.c e70/56 ondisk, result = 0)", 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.318750131999998, 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051419185999999999, 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807681+0000", 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807682+0000", 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2279999999999999e-06 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807687+0000", 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0749999999999997e-06 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807704+0000", 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7424000000000001e-05 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.807714+0000", 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3400000000000004e-06 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858714+0000", 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051000248999999998 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.858732+0000", 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7995999999999998e-05 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:45.859100+0000", 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036787400000000002 2024-04-24T00:35:47.175 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35694 3.1b 3:dcf0eb61:::rbd_data.115ec5833ea9.00000000000000aa:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.206934281000002, 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049399446, 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35694 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919497+0000", 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919498+0000", 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1400000000000001e-06 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919504+0000", 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9170000000000003e-06 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919520+0000", 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5908000000000001e-05 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919532+0000", 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2964e-05 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919630+0000", 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7096999999999995e-05 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.919734+0000", 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104128 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.922056+0000", 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023219149999999999 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968775+0000", 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046719274999999998 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968817+0000", 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-05 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968858+0000", 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1440999999999998e-05 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.968896+0000", 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7741000000000003e-05 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.181 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.235601516999999, 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:35:47.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44664 3.16 3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 524288~131072 [fadvise_sequential+fadvise_nocache],write 1179648~1048576 [fadvise_sequential+fadvise_nocache],write 10616832~2048 [fadvise_sequential+fadvise_nocache],write 10627072~4096 [fadvise_sequential+fadvise_nocache],write 10639360~4096 [fadvise_sequential+fadvise_nocache],write 10651648~4096 [fadvise_sequential+fadvise_nocache],write 10663936~4096 [fadvise_sequential+fadvise_nocache],write 10676224~4096 [fadvise_sequential+fadvise_nocache],write 10688512~4096 [fadvise_sequential+fadvise_nocache],write 10700800~4096 [fadvise_sequential+fadvise_nocache],write 10713088~4096 [fadvise_sequential+fadvise_nocache],write 10725376~4096 [fadvise_sequential+fadvise_nocache],write 10737664~4096 [fadvise_sequential+fadvise_nocache],write 10754048~4096 [fadvise_sequential+fadvise_nocache],write 10766336~4096 [fadvise_sequential+fadvise_nocache],write 10778624~4096 [fadvise_sequential+fadvise_nocache],write 10790912~4096 [fadvise_sequential+fadvise_nocache],write 10803200~4096 [fadvise_sequential+fadvise_nocache],write 10815488~4096 [fadvise_sequential+fadvise_nocache],write 10827776~4096 [fadvise_sequential+fadvise_nocache],write 10840064~4096 [fadvise_sequential+fadvise_nocache],write 10852352~4096 [fadvise_sequential+fadvise_nocache],write 10864640~4096 [fadvise_sequential+fadvise_nocache],write 10876928~2048 [fadvise_sequential+fadvise_nocache],write 10881024~4096 [fadvise_sequential+fadvise_nocache],write 10893312~4096 [fadvise_sequential+fadvise_nocache],write 10905600~4096 [fadvise_sequential+fadvise_nocache],write 10917888~4096 [fadvise_sequential+fadvise_nocache],write 10930176~4096 [fadvise_sequential+fadvise_nocache],write 10942464~4096 [fadvise_sequential+fadvise_nocache],write 10954752~4096 [fadvise_sequential+fadvise_nocache],write 10967040~4096 [fadvise_sequential+fadvise_nocache],write 10979328~4096 [fadvise_sequential+fadvise_nocache],write 10991616~4096 [fadvise_sequential+fadvise_nocache],write 11003904~4096 [fadvise_sequential+fadvise_nocache],write 11010048~2048 [fadvise_sequential+fadvise_nocache],write 11020288~4096 [fadvise_sequential+fadvise_nocache],write 11032576~4096 [fadvise_sequential+fadvise_nocache],write 11044864~4096 [fadvise_sequential+fadvise_nocache],write 11057152~4096 [fadvise_sequential+fadvise_nocache],write 11069440~4096 [fadvise_sequential+fadvise_nocache],write 11081728~4096 [fadvise_sequential+fadvise_nocache],write 11094016~4096 [fadvise_sequential+fadvise_nocache],write 11106304~4096 [fadvise_sequential+fadvise_nocache],write 11118592~4096 [fadvise_sequential+fadvise_nocache],write 11130880~4096 [fadvise_sequential+fadvise_nocache],write 14417920~4096 [fadvise_sequential+fadvise_nocache],write 14426112~8192 [fadvise_sequential+fadvise_nocache],write 14438400~8192 [fadvise_sequential+fadvise_nocache],write 14450688~8192 [fadvise_sequential+fadvise_nocache],write 14462976~8192 [fadvise_sequential+fadvise_nocache],write 14475264~8192 [fadvise_sequential+fadvise_nocache],write 14487552~8192 [fadvise_sequential+fadvise_nocache],write 14499840~8192 [fadvise_sequential+fadvise_nocache],write 14512128~8192 [fadvise_sequential+fadvise_nocache],write 14524416~8192 [fadvise_sequential+fadvise_nocache],write 14536704~8192 [fadvise_sequential+fadvise_nocache],write 14553088~8192 [fadvise_sequential+fadvise_nocache],write 14565376~8192 [fadvise_sequential+fadvise_nocache],write 14577664~8192 [fadvise_sequential+fadvise_nocache],write 14589952~8192 [fadvise_sequential+fadvise_nocache],write 14602240~8192 [fadvise_sequential+fadvise_nocache],write 14614528~8192 [fadvise_sequential+fadvise_nocache],write 14626816~8192 [fadvise_sequential+fadvise_nocache],write 14639104~8192 [fadvise_sequential+fadvise_nocache],write 14651392~8192 [fadvise_sequential+fadvise_nocache],write 14663680~8192 [fadvise_sequential+fadvise_nocache],write 14675968~4096 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.223528195999997, 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768734, 2024-04-24T00:35:47.187 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44664 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902903+0000", 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.902904+0000", 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5179999999999999e-06 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.904804+0000", 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018996390000000001 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905393+0000", 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00058872600000000003 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905408+0000", 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5073e-05 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905482+0000", 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4284999999999998e-05 2024-04-24T00:35:47.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.905785+0000", 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00030266100000000002 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.907249+0000", 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001464899 2024-04-24T00:35:47.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.932396+0000", 2024-04-24T00:35:47.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025146379 2024-04-24T00:35:47.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958308+0000", 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025912418999999999 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958527+0000", 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000218297 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.958671+0000", 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144838 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.647841378000003, 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.194 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:35:47.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:35:47.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.622572771999998, 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:67305 3.17 e89/59 ondisk, result = 0)", 2024-04-24T00:35:47.204 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.271798695999998, 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049974565999999998, 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854632+0000", 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854633+0000", 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854637+0000", 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.946e-06 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854649+0000", 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2507000000000001e-05 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.854658+0000", 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6330000000000002e-06 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904468+0000", 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049809535000000002 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904479+0000", 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.17e-05 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:08.904607+0000", 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012738300000000001 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.208 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.769369052000002, 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:35:47.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.736763937999999, 2024-04-24T00:35:47.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:35:47.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.332189243999998, 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:35:47.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:35:47.223 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.274349013999998, 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:35:47.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:35:47.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:35:47.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81512 3.5 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.163215879999999, 2024-04-24T00:35:47.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061553231999999999, 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81512 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.229 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963215+0000", 2024-04-24T00:35:47.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963217+0000", 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9439999999999999e-06 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967964+0000", 2024-04-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047468529999999997 2024-04-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967986+0000", 2024-04-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1852e-05 2024-04-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968000+0000", 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4426e-05 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968069+0000", 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9338000000000006e-05 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968219+0000", 2024-04-24T00:35:47.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014914999999999999 2024-04-24T00:35:47.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:47.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.968363+0000", 2024-04-24T00:35:47.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000144093 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993999+0000", 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025636744999999999 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024681+0000", 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030681452000000001 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024722+0000", 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1115999999999997e-05 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.024768+0000", 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6263000000000001e-05 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84133 3.1f 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head [write 0~4030464 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 12.953965527999999, 2024-04-24T00:35:47.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060722168, 2024-04-24T00:35:47.238 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.238 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.238 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.238 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84133 2024-04-24T00:35:47.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172465+0000", 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172468+0000", 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2869999999999999e-06 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180251+0000", 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0077834189999999998 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180276+0000", 2024-04-24T00:35:47.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4932000000000001e-05 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180288+0000", 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2224e-05 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180378+0000", 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9401999999999995e-05 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180542+0000", 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016484599999999999 2024-04-24T00:35:47.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180700+0000", 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015754300000000001 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.208480+0000", 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027780143 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233097+0000", 2024-04-24T00:35:47.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02461675 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233151+0000", 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4593999999999999e-05 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.233187+0000", 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6028000000000001e-05 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.243 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:47.245 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:47.245 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:47.239+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:47.346 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:47.343+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:47.414 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:47.414 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:47.414 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:47.414 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:47.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:47.447 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:47.441+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:47.548 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:47.545+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:47.604 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:47.604 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:47.604 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:47.604 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:47.605 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:47.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:47.662 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:47.659+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:47.738 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:47.738 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:47.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:47.738 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.201747884, 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:35:47.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:35:47.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:47.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:47.744 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.167710830999994, 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:47.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:35:47.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:35:47.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.576984866999993, 2024-04-24T00:35:47.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:35:47.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.567728742, 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:35:47.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:35:47.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:47.757 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.052596835000003, 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:47.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:35:47.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.025146874000001, 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:35:47.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:42868 3.1e 3:781b6cfc:::rbd_data.115e56ad4d37.000000000000003c:snapdir [list-snaps] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.980445668999998, 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021882233000000001, 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 42868 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753558+0000", 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6869999999999999e-06 2024-04-24T00:35:47.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753573+0000", 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.446e-05 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753615+0000", 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.227e-05 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753633+0000", 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8240000000000002e-05 2024-04-24T00:35:47.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.754227+0000", 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059412099999999997 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.775439+0000", 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021211455000000001 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72671 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.082165213, 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017794547000000001, 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72671 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651841+0000", 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651856+0000", 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5549999999999999e-05 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651907+0000", 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1187000000000003e-05 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651933+0000", 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5550000000000001e-05 2024-04-24T00:35:47.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652300+0000", 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036690699999999998 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652318+0000", 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8315999999999999e-05 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669396+0000", 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017078233000000002 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669631+0000", 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000235028 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72679 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.081988101, 2024-04-24T00:35:47.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019600632999999999, 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72679 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652016+0000", 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652025+0000", 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8729999999999996e-06 2024-04-24T00:35:47.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652047+0000", 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2957000000000001e-05 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652065+0000", 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7289999999999999e-05 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652773+0000", 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070851400000000004 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652791+0000", 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7649999999999999e-05 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671513+0000", 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018722486 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671615+0000", 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010119 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72759 3.a 3:541a2ac3:::rbd_data.115efb339e04.000000000000006f:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.080379304000001, 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017161025999999999, 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:47.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72759 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653624+0000", 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653629+0000", 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1529999999999996e-06 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653660+0000", 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0623999999999997e-05 2024-04-24T00:35:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653718+0000", 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8514999999999998e-05 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654870+0000", 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001151909 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654881+0000", 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.118e-05 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670564+0000", 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015682413999999999 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670784+0000", 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022022799999999999 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72761 3.a 3:50a1eac1:::rbd_data.115efb339e04.0000000000000072:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:47.784 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.080176963, 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017246102999999999, 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72761 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:47.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653826+0000", 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9699999999999994e-07 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653843+0000", 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7240000000000001e-05 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653857+0000", 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3823e-05 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653868+0000", 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1055e-05 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654965+0000", 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001097085 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654987+0000", 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1759999999999998e-05 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670852+0000", 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015864494999999999 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671071+0000", 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021964799999999999 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72776 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.080099313000002, 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017442424000000002, 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:47.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72776 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:47.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653904+0000", 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1200000000000001e-06 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653914+0000", 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7669999999999998e-06 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653928+0000", 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4165000000000001e-05 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653940+0000", 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-05 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655028+0000", 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010878579999999999 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:47.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655033+0000", 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9339999999999997e-06 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671139+0000", 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016105922000000002 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671345+0000", 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020662799999999999 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.793 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72783 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.080045738999999, 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017776166999999999, 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72783 2024-04-24T00:35:47.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653957+0000", 2024-04-24T00:35:47.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1699999999999997e-07 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653962+0000", 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6079999999999998e-06 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653975+0000", 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2974e-05 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653986+0000", 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0851e-05 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654503+0000", 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051748099999999997 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654523+0000", 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0032999999999999e-05 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671644+0000", 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017121032000000001 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671733+0000", 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8270999999999996e-05 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:81511 3.d e111/106 3:b780edbf:::rbd_data.115e6f07caf6.0000000000000006:head v 111'118, mlcod=0'0)", 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.770150556000001, 2024-04-24T00:35:47.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029826869999999998, 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.65e-07 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966954+0000", 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031022379999999998 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966976+0000", 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1565000000000001e-05 2024-04-24T00:35:47.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966988+0000", 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2428e-05 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967105+0000", 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011725799999999999 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967132+0000", 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6863999999999999e-05 2024-04-24T00:35:47.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993615+0000", 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026482946 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993678+0000", 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3206000000000004e-05 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84130 3.11 e111/106 3:8ac0652c:::rbd_data.115e6f07caf6.000000000000000b:head v 111'344, mlcod=108'342)", 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.563607529, 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037411427999999997, 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0699999999999999e-07 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171887+0000", 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00149255 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171909+0000", 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1270000000000001e-05 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171918+0000", 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6139999999999998e-06 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172147+0000", 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022887100000000001 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172341+0000", 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019398700000000001 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207639+0000", 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035298445999999997 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207806+0000", 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016648300000000001 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84132 3.14 e111/106 3:2c6d4ac0:::rbd_data.115e6f07caf6.000000000000000d:head v 111'167, mlcod=0'0)", 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.553773669, 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029744291999999999, 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:47.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180230+0000", 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8029999999999999e-06 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.184974+0000", 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047440540000000002 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185003+0000", 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8860000000000002e-05 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185021+0000", 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7473999999999999e-05 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185169+0000", 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148305 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185196+0000", 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7444e-05 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209925+0000", 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024728208000000002 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209973+0000", 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8143999999999998e-05 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.105075041999999, 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.811 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.098803783999999, 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.815 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:47.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:35:47.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.087349198, 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:35:47.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:35:47.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:35:47.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.032896654, 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:35:47.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:35:47.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:47.829 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:47.831 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:47.831 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:47.827+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:47.932 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:47.929+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:48.001 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:48.002 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:48.002 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:48.002 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:48.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:48.033 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:48.025+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:48.134 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:48.131+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:48.137 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:48.137 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:48.137 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:48.137 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:48.137 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:48.151 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:48.238 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:48.235+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.38714521199999, 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.318 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:35:48.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.455179524000002, 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.324 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:35:48.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:48.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.329 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.453930775999993, 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:48.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:35:48.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:35:48.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:35:48.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.335 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 85.296306672, 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:48.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:35:48.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:35:48.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:35:48.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 85.282812129000007, 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:35:48.341 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.021031014000002, 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:48.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:48.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.350 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.767444870999995, 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:35:48.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:48.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.356 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.357 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:48.357 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:48.357 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.739771007000002, 2024-04-24T00:35:48.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:35:48.357 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.358 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.358 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.359 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:35:48.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.359 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:48.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:35:48.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:35:48.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:35:48.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:35:48.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.245748187000004, 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:35:48.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:35:48.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:35:48.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:35:48.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:48.371 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.236739353000004, 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:35:48.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:35:48.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.735040404999999, 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.376 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:35:48.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.734978058999999, 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:35:48.380 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.734468380000003, 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:35:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.734357318000001, 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.389 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:35:48.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:35:48.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:35:48.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:35:48.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:35:48.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.395 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.395 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.395 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.395 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:48.395 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.734292052000001, 2024-04-24T00:35:48.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:35:48.396 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.396 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.396 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.396 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.396 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.396 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.396 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:35:48.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:48.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:35:48.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:35:48.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:35:48.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:35:48.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:35:48.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:35:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:35:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:35:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:35:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:35:48.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:35:48.402 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.402 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.402 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.403 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.403 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:48.403 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.734231985999998, 2024-04-24T00:35:48.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:35:48.403 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.403 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.403 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.403 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.404 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:35:48.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.404 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:48.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:48.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:35:48.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:35:48.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:35:48.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.734175293, 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.409 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:35:48.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.734105960000001, 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.414 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.417 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.734015341000003, 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:35:48.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:35:48.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.421 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.733971928000003, 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:48.422 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:35:48.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.425 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:48.426 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:48.426 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:48.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:48.430 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:48.421+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:48.528 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:48.523+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:48.566 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:48.567 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:48.567 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:48.567 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:48.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:48.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:35:48.630 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:48.625+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:48.656 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:35:48.656 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:35:48.657 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-24T00:35:48.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-04-24T00:35:48.731 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:48.727+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:48.760 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:48.760 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:48.760 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:48.760 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:48.760 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:48.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:48.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:48.831 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:48.827+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:48.932 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:48.927+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:48.938 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:48.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:48.950 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:48.950 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:48.956 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:48.956 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:48.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:48.990 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:48.990 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:49.023 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:49.023 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:49.033 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:49.027+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:49.059 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:49.059 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:49.081 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:49.081 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:49.084 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:49.084 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:49.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:49.101 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:49.102 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:49.122 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:49.123 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:49.134 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:49.129+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:49.219 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:49.219 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:49.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:49.235 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:49.229+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:49.336 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:49.331+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:49.358 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:49.358 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:49.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:49.437 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:49.435+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:49.492 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:49.492 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:49.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:49.554 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:49.549+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:49.599 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:35:49.603 INFO:teuthology.orchestra.run.smithi064.stdout:moved 4 / 82 (4.87805%) 2024-04-24T00:35:49.603 INFO:teuthology.orchestra.run.smithi064.stdout:avg 13.6667 2024-04-24T00:35:49.603 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 5.58769 -> 6.01849 (expected baseline 3.37474) 2024-04-24T00:35:49.603 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.1 with 2 -> 2 pgs (0.146341 -> 0.146341 * mean) 2024-04-24T00:35:49.604 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.4 with 19 -> 20 pgs (1.39024 -> 1.46341 * mean) 2024-04-24T00:35:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:35:49.604 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-04-24T00:35:49.604 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 3 2024-04-24T00:35:49.604 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-04-24T00:35:49.604 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0005 2024-04-24T00:35:49.604 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0005 2024-04-24T00:35:49.604 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 weight 1.0000 -> 0.7930 2024-04-24T00:35:49.638 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-24T00:35:49.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-04-24T00:35:49.640 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:49.635+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:49.643 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:49.643 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:49.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:49.701 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-04-24T00:35:49.740 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:49.735+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:49.822 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:49.822 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:49.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:49.882 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:49.877+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:49.942 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:49.939+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:49.949 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:49.949 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:49.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:49.964 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:35:49.959+0000 7f43e5c71640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-04-24T00:35:49.964 INFO:teuthology.orchestra.run.smithi064.stdout:moved 0 / 82 (0%) 2024-04-24T00:35:49.964 INFO:teuthology.orchestra.run.smithi064.stdout:avg 13.6667 2024-04-24T00:35:49.964 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 6.01849 -> 6.01849 (expected baseline 3.37474) 2024-04-24T00:35:49.964 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.1 with 2 -> 2 pgs (0.146341 -> 0.146341 * mean) 2024-04-24T00:35:49.964 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.4 with 20 -> 20 pgs (1.46341 -> 1.46341 * mean) 2024-04-24T00:35:49.964 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:35:49.964 INFO:teuthology.orchestra.run.smithi064.stdout:oload 1000 2024-04-24T00:35:49.965 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 0.05 2024-04-24T00:35:49.965 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-04-24T00:35:49.965 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0005 2024-04-24T00:35:49.965 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0052 2024-04-24T00:35:49.965 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 weight 0.2503 -> 0.2642 2024-04-24T00:35:49.965 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-04-24T00:35:49.983 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-24T00:35:49.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-04-24T00:35:50.073 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:50.069+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:50.095 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:50.096 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:50.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:50.154 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:50.149+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:50.244 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:50.237+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:50.268 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:50.268 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:50.268 INFO:teuthology.orchestra.run.smithi064.stdout:3429 read 0x3f9b412 thru 0x3fa230a (0x6ef9 bytes) 2024-04-24T00:35:50.269 INFO:teuthology.orchestra.run.smithi064.stdout:3430 writesame 0x1cccf3 thru 0x1dab6d data_size 0x5(0xde7b bytes) 2024-04-24T00:35:50.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:50.323 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:35:50.319+0000 7f43e5c71640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:moved 0 / 82 (0%) 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:avg 13.6667 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 6.01849 -> 6.01849 (expected baseline 3.37474) 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.1 with 2 -> 2 pgs (0.146341 -> 0.146341 * mean) 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.4 with 20 -> 20 pgs (1.46341 -> 1.46341 * mean) 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:oload 1000 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 1 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-04-24T00:35:50.325 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0005 2024-04-24T00:35:50.326 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0052 2024-04-24T00:35:50.326 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 weight 0.2503 -> 0.2642 2024-04-24T00:35:50.326 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-04-24T00:35:50.338 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:50.341 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-24T00:35:50.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-04-24T00:35:50.345 INFO:teuthology.orchestra.run.smithi064.stdout:3432 trunc from 0x5574027 to 0xb12f134 2024-04-24T00:35:50.346 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:50.341+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:50.354 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:50.355 INFO:teuthology.orchestra.run.smithi064.stdout:3434 read 0x6d62f84 thru 0x6d6faea (0xcb67 bytes) 2024-04-24T00:35:50.356 INFO:teuthology.orchestra.run.smithi064.stdout:3435 read 0x7c0fba3 thru 0x7c166b5 (0x6b13 bytes) 2024-04-24T00:35:50.356 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:50.356 INFO:teuthology.orchestra.run.smithi064.stdout:3439 punch from 0x98c9193 to 0x98ceb8a, (0x59f7 bytes) 2024-04-24T00:35:50.356 INFO:teuthology.orchestra.run.smithi064.stdout:3440 writesame 0x1d1fb48 thru 0x1d2992b data_size 0xac(0x9de4 bytes) 2024-04-24T00:35:50.359 INFO:teuthology.orchestra.run.smithi064.stdout:3441 write 0x9cd1381 thru 0x9cd9a74 (0x86f4 bytes) 2024-04-24T00:35:50.411 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:50.411 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:50.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:50.471 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:50.467+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:50.547 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:50.543+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:50.595 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:50.595 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:50.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:50.644 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-04-24T00:35:50.639+0000 7f43e5c71640 -1 mgr.server reply reply (2) No such file or directory no change 2024-04-24T00:35:50.645 INFO:teuthology.orchestra.run.smithi064.stdout:moved 0 / 82 (0%) 2024-04-24T00:35:50.645 INFO:teuthology.orchestra.run.smithi064.stdout:avg 13.6667 2024-04-24T00:35:50.645 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 6.01849 -> 6.01849 (expected baseline 3.37474) 2024-04-24T00:35:50.645 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.1 with 2 -> 2 pgs (0.146341 -> 0.146341 * mean) 2024-04-24T00:35:50.645 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.4 with 20 -> 20 pgs (1.46341 -> 1.46341 * mean) 2024-04-24T00:35:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:35:50.645 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-04-24T00:35:50.646 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 3 2024-04-24T00:35:50.646 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-04-24T00:35:50.646 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0005 2024-04-24T00:35:50.646 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0006 2024-04-24T00:35:50.646 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 weight 0.7930 -> 0.6612 2024-04-24T00:35:50.646 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 weight 0.2503 -> 0.2642 2024-04-24T00:35:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-04-24T00:35:50.648 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:50.643+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:50.662 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-04-24T00:35:50.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-04-24T00:35:50.749 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:50.743+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:50.767 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:50.767 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:50.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:50.870 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:50.867+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:50.950 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:50.950 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:50.951 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:50.945+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:50.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:51.055 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:51.051+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:51.132 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:51.132 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:51.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:51.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:51.192 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:51.187+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:51.252 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:51.245+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:51.298 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:51.298 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:51.314 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:51.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:51.315 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:51.316 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:51.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:51.331 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:51.331 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:51.346 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:51.346 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:51.352 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:51.345+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:51.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:51.364 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:51.381 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:51.382 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:51.402 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:51.402 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:51.420 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:51.421 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:51.450 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:51.450 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:51.451 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:51.451 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:51.454 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:51.447+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:51.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:51.554 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:51.549+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:51.612 INFO:teuthology.orchestra.run.smithi064.stdout:moved 0 / 82 (0%) 2024-04-24T00:35:51.612 INFO:teuthology.orchestra.run.smithi064.stdout:avg 13.6667 2024-04-24T00:35:51.612 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 6.01849 -> 6.01849 (expected baseline 3.37474) 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.1 with 2 -> 2 pgs (0.146341 -> 0.146341 * mean) 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.4 with 20 -> 20 pgs (1.46341 -> 1.46341 * mean) 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:oload 1000 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 3 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0005 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0052 2024-04-24T00:35:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 weight 0.2503 -> 0.2642 2024-04-24T00:35:51.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:35:51.635 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:51.635 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:51.636 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:51.636 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:51.636 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:51.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:51.694 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:51.691+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:51.756 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:51.749+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:51.778 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:51.778 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:51.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:51.778 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:51.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.778 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 93.241173712999995, 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:35:51.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:51.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:35:51.784 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 93.207136660000003, 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:51.788 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.616410696000003, 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:35:51.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:35:51.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.607154570999995, 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.794 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:35:51.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:35:51.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.092022663999998, 2024-04-24T00:35:51.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.803 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.064572703000003, 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:35:51.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:42868 3.1e 3:781b6cfc:::rbd_data.115e56ad4d37.000000000000003c:snapdir [list-snaps] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.019871498000001, 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021882233000000001, 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.808 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 42868 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753556+0000", 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753558+0000", 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6869999999999999e-06 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753573+0000", 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.446e-05 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753615+0000", 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.227e-05 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.753633+0000", 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8240000000000002e-05 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.754227+0000", 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00059412099999999997 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.775439+0000", 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021211455000000001 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.811 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72671 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.121591041999999, 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017794547000000001, 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72671 2024-04-24T00:35:51.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651837+0000", 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651841+0000", 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651856+0000", 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5549999999999999e-05 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651907+0000", 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1187000000000003e-05 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.651933+0000", 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5550000000000001e-05 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652300+0000", 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036690699999999998 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652318+0000", 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8315999999999999e-05 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669396+0000", 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017078233000000002 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.669631+0000", 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000235028 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.816 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72679 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.121413929999999, 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019600632999999999, 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72679 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652014+0000", 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652016+0000", 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6729999999999999e-06 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652025+0000", 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8729999999999996e-06 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652047+0000", 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2957000000000001e-05 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652065+0000", 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7289999999999999e-05 2024-04-24T00:35:51.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652773+0000", 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00070851400000000004 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.652791+0000", 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7649999999999999e-05 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671513+0000", 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018722486 2024-04-24T00:35:51.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671615+0000", 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010119 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72759 3.a 3:541a2ac3:::rbd_data.115efb339e04.000000000000006f:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.119805133, 2024-04-24T00:35:51.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017161025999999999, 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72759 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:51.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653623+0000", 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653624+0000", 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.003e-06 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653629+0000", 2024-04-24T00:35:51.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1529999999999996e-06 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653660+0000", 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0623999999999997e-05 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653718+0000", 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8514999999999998e-05 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654870+0000", 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001151909 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654881+0000", 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.118e-05 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670564+0000", 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015682413999999999 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670784+0000", 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022022799999999999 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72761 3.a 3:50a1eac1:::rbd_data.115efb339e04.0000000000000072:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.119602791999998, 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017246102999999999, 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.826 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72761 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653825+0000", 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653826+0000", 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9699999999999994e-07 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653843+0000", 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7240000000000001e-05 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653857+0000", 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3823e-05 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653868+0000", 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1055e-05 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654965+0000", 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001097085 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654987+0000", 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1759999999999998e-05 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.670852+0000", 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015864494999999999 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671071+0000", 2024-04-24T00:35:51.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021964799999999999 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72776 3.a 3:5408e9e2:::rbd_data.115efb339e04.000000000000000a:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.119525142000001, 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017442424000000002, 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72776 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653903+0000", 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653904+0000", 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1200000000000001e-06 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653914+0000", 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7669999999999998e-06 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653928+0000", 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4165000000000001e-05 2024-04-24T00:35:51.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653940+0000", 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-05 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655028+0000", 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010878579999999999 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.655033+0000", 2024-04-24T00:35:51.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9339999999999997e-06 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671139+0000", 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016105922000000002 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671345+0000", 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020662799999999999 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72783 3.1e 3:7da706d8:::rbd_data.115efb339e04.0000000000000012:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 31.119471568000002, 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017776166999999999, 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72783 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653956+0000", 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653957+0000", 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1699999999999997e-07 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653962+0000", 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6079999999999998e-06 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653975+0000", 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2974e-05 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.653986+0000", 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0851e-05 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654503+0000", 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00051748099999999997 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting_for_map not empty", 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.654523+0000", 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0032999999999999e-05 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671644+0000", 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017121032000000001 2024-04-24T00:35:51.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:20.671733+0000", 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8270999999999996e-05 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:81511 3.d e111/106 3:b780edbf:::rbd_data.115e6f07caf6.0000000000000006:head v 111'118, mlcod=0'0)", 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.809576385, 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029826869999999998, 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.840 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.65e-07 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966954+0000", 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031022379999999998 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966976+0000", 2024-04-24T00:35:51.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1565000000000001e-05 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966988+0000", 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2428e-05 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967105+0000", 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011725799999999999 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967132+0000", 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6863999999999999e-05 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993615+0000", 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026482946 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993678+0000", 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3206000000000004e-05 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84130 3.11 e111/106 3:8ac0652c:::rbd_data.115e6f07caf6.000000000000000b:head v 111'344, mlcod=108'342)", 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.603033358000001, 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037411427999999997, 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0699999999999999e-07 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171887+0000", 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00149255 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171909+0000", 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1270000000000001e-05 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171918+0000", 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6139999999999998e-06 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172147+0000", 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022887100000000001 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172341+0000", 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019398700000000001 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207639+0000", 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035298445999999997 2024-04-24T00:35:51.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207806+0000", 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016648300000000001 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84132 3.14 e111/106 3:2c6d4ac0:::rbd_data.115e6f07caf6.000000000000000d:head v 111'167, mlcod=0'0)", 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.593199498000001, 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029744291999999999, 2024-04-24T00:35:51.849 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180230+0000", 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8029999999999999e-06 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.184974+0000", 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047440540000000002 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185003+0000", 2024-04-24T00:35:51.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8860000000000002e-05 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185021+0000", 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7473999999999999e-05 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185169+0000", 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148305 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185196+0000", 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7444e-05 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209925+0000", 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024728208000000002 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209973+0000", 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8143999999999998e-05 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.144500871, 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:35:51.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.138229613, 2024-04-24T00:35:51.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:51.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.126775027000001, 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.864 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.072322483000001, 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:51.868 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:35:51.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:35:51.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:51.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:51.872 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:51.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:51.874 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:51.869+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:51.925 INFO:teuthology.orchestra.run.smithi064.stdout:3442 write 0x81df0f8 thru 0x81e2d25 (0x3c2e bytes) 2024-04-24T00:35:51.935 INFO:teuthology.orchestra.run.smithi064.stdout:3443 writesame 0x4194b2b thru 0x4196ba1 data_size 0x1(0x2077 bytes) 2024-04-24T00:35:51.942 INFO:teuthology.orchestra.run.smithi064.stdout:3444 read 0xe1589f thru 0xe1ca90 (0x71f2 bytes) 2024-04-24T00:35:51.943 INFO:teuthology.orchestra.run.smithi064.stdout:3446 punch from 0x75faa61 to 0x7605646, (0xabe5 bytes) 2024-04-24T00:35:51.943 INFO:teuthology.orchestra.run.smithi064.stdout:3447 write 0x66c77b5 thru 0x66c7d30 (0x57c bytes) 2024-04-24T00:35:51.947 INFO:teuthology.orchestra.run.smithi064.stdout:3448 read 0x8bcdb50 thru 0x8bdbffb (0xe4ac bytes) 2024-04-24T00:35:51.947 INFO:teuthology.orchestra.run.smithi064.stdout:3449 punch from 0xa27b48d to 0xa286652, (0xb1c5 bytes) 2024-04-24T00:35:51.947 INFO:teuthology.orchestra.run.smithi064.stdout:3450 punch from 0x625d902 to 0x6269cc7, (0xc3c5 bytes) 2024-04-24T00:35:51.947 INFO:teuthology.orchestra.run.smithi064.stdout:3452 punch from 0x1ffe6e3 to 0x200d186, (0xeaa3 bytes) 2024-04-24T00:35:51.948 INFO:teuthology.orchestra.run.smithi064.stdout:3453 trunc from 0xb12f134 to 0xa205c68 2024-04-24T00:35:51.962 INFO:teuthology.orchestra.run.smithi064.stdout:3456 writesame 0x1ee5338 thru 0x1eecfbf data_size 0xa(0x7c88 bytes) 2024-04-24T00:35:51.970 INFO:teuthology.orchestra.run.smithi064.stdout:3457 write 0x1860f58 thru 0x1866f28 (0x5fd1 bytes) 2024-04-24T00:35:51.975 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:51.971+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:51.976 INFO:teuthology.orchestra.run.smithi064.stdout:3458 writesame 0x928000b thru 0x928a464 data_size 0x2(0xa45a bytes) 2024-04-24T00:35:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:3459 write 0xb7d519c thru 0xb7d5b2b (0x990 bytes) 2024-04-24T00:35:52.049 INFO:teuthology.orchestra.run.smithi064.stdout:3460 write 0x4517d46 thru 0x451e7a7 (0x6a62 bytes) 2024-04-24T00:35:52.055 INFO:teuthology.orchestra.run.smithi064.stdout:3461 write 0x471dd4d thru 0x4720b0b (0x2dbf bytes) 2024-04-24T00:35:52.057 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:52.057 INFO:teuthology.orchestra.run.smithi064.stdout:3463 write 0x15d490d thru 0x15e48a1 (0xff95 bytes) 2024-04-24T00:35:52.068 INFO:teuthology.orchestra.run.smithi064.stdout:3464 write 0x9062d6c thru 0x906d75d (0xa9f2 bytes) 2024-04-24T00:35:52.075 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:52.075 INFO:teuthology.orchestra.run.smithi064.stdout:3466 punch from 0xe90296 to 0xe96e56, (0x6bc0 bytes) 2024-04-24T00:35:52.075 INFO:teuthology.orchestra.run.smithi064.stdout:3468 read 0x5ac106c thru 0x5ac9d0a (0x8c9f bytes) 2024-04-24T00:35:52.076 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:52.069+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:52.076 INFO:teuthology.orchestra.run.smithi064.stdout:3469 read 0x60f6ca7 thru 0x6105c6e (0xefc8 bytes) 2024-04-24T00:35:52.076 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:52.076 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:52.076 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:52.076 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:52.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:52.096 INFO:teuthology.orchestra.run.smithi064.stdout:3470 write 0xdf65ef1 thru 0xdf6ea05 (0x8b15 bytes) 2024-04-24T00:35:52.099 INFO:teuthology.orchestra.run.smithi064.stdout:3471 writesame 0xcbaf257 thru 0xcbb2f55 data_size 0xf(0x3cff bytes) 2024-04-24T00:35:52.105 INFO:teuthology.orchestra.run.smithi064.stdout:3472 write 0x6fde57e thru 0x6fe06c6 (0x2149 bytes) 2024-04-24T00:35:52.109 INFO:teuthology.orchestra.run.smithi064.stdout:3474 write 0x7c9171e thru 0x7c9c5bf (0xaea2 bytes) 2024-04-24T00:35:52.122 INFO:teuthology.orchestra.run.smithi064.stdout:3476 write 0x8980751 thru 0x8983441 (0x2cf1 bytes) 2024-04-24T00:35:52.122 INFO:teuthology.orchestra.run.smithi064.stdout:3478 read 0x1fb3cab thru 0x1fb4297 (0x5ed bytes) 2024-04-24T00:35:52.123 INFO:teuthology.orchestra.run.smithi064.stdout:3479 write 0x33a06a thru 0x341dee (0x7d85 bytes) 2024-04-24T00:35:52.129 INFO:teuthology.orchestra.run.smithi064.stdout:3483 read 0x6812222 thru 0x681313d (0xf1c bytes) 2024-04-24T00:35:52.130 INFO:teuthology.orchestra.run.smithi064.stdout:3484 punch from 0x9bd4d42 to 0x9be0304, (0xb5c2 bytes) 2024-04-24T00:35:52.157 INFO:teuthology.orchestra.run.smithi064.stdout:3485 trunc from 0xdf6ea06 to 0x420cace 2024-04-24T00:35:52.182 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:52.179+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:52.182 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:52.183 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:52.183 INFO:teuthology.orchestra.run.smithi064.stdout:3488 punch from 0x21a12e0 to 0x21a85a9, (0x72c9 bytes) 2024-04-24T00:35:52.183 INFO:teuthology.orchestra.run.smithi064.stdout:3489 read 0x22cb02a thru 0x22d5d43 (0xad1a bytes) 2024-04-24T00:35:52.188 INFO:teuthology.orchestra.run.smithi064.stdout:3490 trunc from 0x420cace to 0x5db122f 2024-04-24T00:35:52.200 INFO:teuthology.orchestra.run.smithi064.stdout:3491 read 0x2afe9e thru 0x2b8976 (0x8ad9 bytes) 2024-04-24T00:35:52.215 INFO:teuthology.orchestra.run.smithi064.stdout:3492 write 0x6757d85 thru 0x67598fb (0x1b77 bytes) 2024-04-24T00:35:52.218 INFO:teuthology.orchestra.run.smithi064.stdout:3493 read 0x4a7eca6 thru 0x4a8d799 (0xeaf4 bytes) 2024-04-24T00:35:52.219 INFO:teuthology.orchestra.run.smithi064.stdout:3494 punch from 0x299ca0a to 0x29a9544, (0xcb3a bytes) 2024-04-24T00:35:52.219 INFO:teuthology.orchestra.run.smithi064.stdout:3496 write 0x6344e16 thru 0x6348906 (0x3af1 bytes) 2024-04-24T00:35:52.239 INFO:teuthology.orchestra.run.smithi064.stdout:3497 trunc from 0x67598fc to 0x171f02 2024-04-24T00:35:52.273 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:52.273 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:52.274 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:52.274 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:52.277 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:52.269+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:52.278 INFO:teuthology.orchestra.run.smithi064.stdout:3498 read 0x15497c thru 0x157c43 (0x32c8 bytes) 2024-04-24T00:35:52.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:52.312 INFO:teuthology.orchestra.run.smithi064.stdout:3499 write 0x7dde07b thru 0x7dde3f5 (0x37b bytes) 2024-04-24T00:35:52.318 INFO:teuthology.orchestra.run.smithi064.stdout:3500 read 0x663d0c0 thru 0x664a4ce (0xd40f bytes) 2024-04-24T00:35:52.318 INFO:teuthology.orchestra.run.smithi064.stdout:3501 read 0x37adc7a thru 0x37afc49 (0x1fd0 bytes) 2024-04-24T00:35:52.325 INFO:teuthology.orchestra.run.smithi064.stdout:3503 trunc from 0x7dde3f6 to 0xa052596 2024-04-24T00:35:52.336 INFO:teuthology.orchestra.run.smithi064.stdout:3504 writesame 0x9d769ee thru 0x9d854b3 data_size 0x12(0xeac6 bytes) 2024-04-24T00:35:52.348 INFO:teuthology.orchestra.run.smithi064.stdout:3506 write 0x8094adf thru 0x80984cc (0x39ee bytes) 2024-04-24T00:35:52.354 INFO:teuthology.orchestra.run.smithi064.stdout:3507 read 0x465a6e6 thru 0x465e4b2 (0x3dcd bytes) 2024-04-24T00:35:52.355 INFO:teuthology.orchestra.run.smithi064.stdout:3508 read 0x13dd965 thru 0x13ea817 (0xceb3 bytes) 2024-04-24T00:35:52.370 INFO:teuthology.orchestra.run.smithi064.stdout:3509 write 0xb2c50de thru 0xb2d0963 (0xb886 bytes) 2024-04-24T00:35:52.375 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:52.379 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:52.373+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:52.397 INFO:teuthology.orchestra.run.smithi064.stdout:3512 write 0xeadb4ea thru 0xeae11c2 (0x5cd9 bytes) 2024-04-24T00:35:52.442 INFO:teuthology.orchestra.run.smithi064.stdout:3513 trunc from 0xeae11c3 to 0x4dc6ea 2024-04-24T00:35:52.478 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:52.478 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:52.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:52.478 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:52.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.478 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:35:52.478 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:52.478 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 131.54844774, 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:35:52.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:35:52.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 98.616482051999995, 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:35:52.485 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:35:52.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:35:52.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:52.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 98.615233304, 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.491 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:35:52.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:35:52.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:52.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.457609199999993, 2024-04-24T00:35:52.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:35:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:35:52.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.444114657, 2024-04-24T00:35:52.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:35:52.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:52.507 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 85.182333541999995, 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:52.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:35:52.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:35:52.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.928747399000002, 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:35:52.513 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:35:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:35:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:35:52.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.901073534999995, 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.519 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:35:52.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:52.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.407050714999997, 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:52.524 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:52.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:35:52.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:35:52.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.398041880999997, 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.530 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:35:52.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:35:52.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.896342933, 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:35:52.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:35:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.896280587, 2024-04-24T00:35:52.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:52.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:35:52.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:35:52.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:35:52.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.895770908000003, 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:35:52.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.895659846000001, 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:35:52.552 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:35:52.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.895594580000001, 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:52.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.895534513999998, 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:35:52.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:35:52.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:35:52.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:35:52.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.895477821, 2024-04-24T00:35:52.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:52.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:35:52.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.895408488000001, 2024-04-24T00:35:52.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:35:52.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.895317869000003, 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:35:52.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:35:52.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.895274456000003, 2024-04-24T00:35:52.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:52.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:35:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:52.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:35:52.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:35:52.585 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.585 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.585 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.585 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:52.585 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:52.585 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:52.587 INFO:teuthology.orchestra.run.smithi064.stdout:3514 write 0xca824df thru 0xca865e9 (0x410b bytes) 2024-04-24T00:35:52.587 INFO:teuthology.orchestra.run.smithi064.stdout:3515 read 0xa3c430d thru 0xa3cd332 (0x9026 bytes) 2024-04-24T00:35:52.587 INFO:teuthology.orchestra.run.smithi064.stdout:3516 read 0x2745e3c thru 0x27533d5 (0xd59a bytes) 2024-04-24T00:35:52.587 INFO:teuthology.orchestra.run.smithi064.stdout:3517 read 0x3026ae9 thru 0x302e091 (0x75a9 bytes) 2024-04-24T00:35:52.587 INFO:teuthology.orchestra.run.smithi064.stdout:3518 writesame 0x58f1ab9 thru 0x58f5789 data_size 0x1(0x3cd1 bytes) 2024-04-24T00:35:52.587 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:52.581+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:52.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:52.595 INFO:teuthology.orchestra.run.smithi064.stdout:3519 write 0xc4a7c87 thru 0xc4b4b38 (0xceb2 bytes) 2024-04-24T00:35:52.607 INFO:teuthology.orchestra.run.smithi064.stdout:3521 write 0x3c15711 thru 0x3c1f671 (0x9f61 bytes) 2024-04-24T00:35:52.612 INFO:teuthology.orchestra.run.smithi064.stdout:3522 read 0x24b4ab1 thru 0x24b829b (0x37eb bytes) 2024-04-24T00:35:52.612 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:35:52.613 INFO:teuthology.orchestra.run.smithi064.stdout:3523 read 0x3185c9 thru 0x32725e (0xec96 bytes) 2024-04-24T00:35:52.627 INFO:teuthology.orchestra.run.smithi064.stdout:3525 writesame 0xdce63ac thru 0xdcea8c6 data_size 0x1(0x451b bytes) 2024-04-24T00:35:52.628 INFO:teuthology.orchestra.run.smithi064.stdout:3526 read 0xc52c19f thru 0xc537427 (0xb289 bytes) 2024-04-24T00:35:52.628 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:52.628 INFO:teuthology.orchestra.run.smithi064.stdout:3529 write 0x9c56e8f thru 0x9c575d1 (0x743 bytes) 2024-04-24T00:35:52.690 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:52.685+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:52.725 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:52.725 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:52.725 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:52.725 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:52.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:52.788 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:52.783+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:52.889 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:52.883+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:52.911 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:52.912 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:52.912 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:52.912 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:52.912 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:52.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:52.990 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:52.985+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:53.053 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:53.053 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:53.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:53.054 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-04-24T00:35:53.054 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:53.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:53.091 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:53.087+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:53.192 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:53.187+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:53.231 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:53.232 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:53.232 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:53.232 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:53.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:53.292 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:53.287+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:53.374 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:53.374 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:53.374 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:53.374 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:53.374 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:53.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:53.434 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:53.429+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:53.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:53.494 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:53.491+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:53.519 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 86.080531757000003, 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:35:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:35:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:35:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:35:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:35:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:35:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 86.025020271000002, 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:35:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:35:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:35:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.924443623000002, 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:35:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:35:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:35:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:35:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:35:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:35:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:35:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.882878668999993, 2024-04-24T00:35:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:35:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:35:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:35:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:35:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:35:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.319147106999999, 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:35:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:35:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:35:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.816173710000001, 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:35:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:35:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:35:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:35:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.577859477000001, 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:35:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:35:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542919053999999, 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:35:53.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542898180999998, 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542874802, 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542853260000001, 2024-04-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:35:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.54283174, 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:35:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:35:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542785583000001, 2024-04-24T00:35:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:35:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542761464000002, 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.592 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.592 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:35:53.592 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542738060000001, 2024-04-24T00:35:53.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:35:53.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.594 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:35:53.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.594 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:35:53.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:35:53.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:35:53.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:35:53.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:35:53.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542710286999998, 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:35:53.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:35:53.603 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542688874, 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.604 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:35:53.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:35:53.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.542668283000001, 2024-04-24T00:35:53.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:35:53.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:35:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.326598041, 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:53.614 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:35:53.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.298627137, 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:35:53.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:53.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:53.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:53.625 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:53.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:53.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:53.627 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:53.628 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:53.628 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:53.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:53.629 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:53.623+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:53.631 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:53.632 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:53.646 INFO:teuthology.orchestra.run.smithi064.stdout:3530 writesame 0xe8ca53d thru 0xe8d695d data_size 0x25(0xc421 bytes) 2024-04-24T00:35:53.646 INFO:teuthology.orchestra.run.smithi064.stdout:3531 read 0x287c267 thru 0x288ad78 (0xeb12 bytes) 2024-04-24T00:35:53.647 INFO:teuthology.orchestra.run.smithi064.stdout:3532 read 0x484b719 thru 0x485ae30 (0xf718 bytes) 2024-04-24T00:35:53.647 INFO:teuthology.orchestra.run.smithi064.stdout:3533 punch from 0x19f0fd0 to 0x19f5452, (0x4482 bytes) 2024-04-24T00:35:53.647 INFO:teuthology.orchestra.run.smithi064.stdout:3537 writesame 0x517a6d4 thru 0x517e074 data_size 0x1(0x39a1 bytes) 2024-04-24T00:35:53.657 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:53.681 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:53.682 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:53.702 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:53.703 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:53.714 INFO:teuthology.orchestra.run.smithi064.stdout:3538 writesame 0xa509894 thru 0xa510656 data_size 0x1(0x6dc3 bytes) 2024-04-24T00:35:53.725 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:53.727 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:53.723+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:53.752 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:53.752 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:53.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:53.801 INFO:teuthology.orchestra.run.smithi064.stdout:3539 punch from 0xe22f120 to 0xe23ab3d, (0xba1d bytes) 2024-04-24T00:35:53.801 INFO:teuthology.orchestra.run.smithi064.stdout:3540 write 0x84558fd thru 0x845d2ed (0x79f1 bytes) 2024-04-24T00:35:53.845 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:53.840+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:53.927 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:53.927 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:53.928 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:53.920+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:53.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:54.030 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:54.024+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:54.111 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:54.112 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:54.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:54.130 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:54.127+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:54.231 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:54.227+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:54.296 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:54.297 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:54.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:54.358 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:54.352+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:54.432 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:54.424+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:54.437 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:54.437 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:54.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:54.542 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:54.536+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:54.578 INFO:teuthology.orchestra.run.smithi064.stdout:3541 clone 24 order 23 su 2097152 sc 2 2024-04-24T00:35:54.633 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:54.633 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:54.634 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:54.631+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:54.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:35:54.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:54.735 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:54.731+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:54.783 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:54.783 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:54.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:54.838 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:54.835+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:54.937 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:54.928+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:54.962 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:54.962 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:54.978 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:55.038 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:55.035+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:55.097 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:55.097 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:55.111 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:55.139 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:55.132+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:55.240 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:55.232+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:55.272 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:55.272 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:55.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:55.341 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:55.335+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:55.406 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:55.406 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:55.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:55.442 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:55.436+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:55.543 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:55.539+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:35:55.579 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:55.579 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:55.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:55.644 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:55.636+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:55.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:35:55.746 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:55.740+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:55.747 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:55.748 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:55.761 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:55.846 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:55.840+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:55.880 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:55.881 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:55.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:35:55.906 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:55.906 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:55.923 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:55.923 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:55.943 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:55.943 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:55.947 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:55.943+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:55.962 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:55.963 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:55.985 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:55.985 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:56.009 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:56.009 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:56.020 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:56.020 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:56.033 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:56.033 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:56.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:35:56.055 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:56.055 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:56.078 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:56.075+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:56.149 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:56.140+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:56.196 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:56.196 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:56.196 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:35:56.196 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:56.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:35:56.254 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:56.251+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:56.329 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:56.329 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:35:56.329 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:35:56.329 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:35:56.329 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:56.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:35:56.351 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:56.344+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:56.452 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:56.447+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:56.513 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:35:56.513 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 135.58252855200001, 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:35:56.514 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:35:56.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:56.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 102.65056286399999, 2024-04-24T00:35:56.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:56.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:35:56.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:35:56.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:56.526 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 102.649314116, 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:35:56.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:35:56.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 93.491690012000007, 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:35:56.532 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 93.478195468999999, 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.538 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.216414353999994, 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.542 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:35:56.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:35:56.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:56.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.962828211000001, 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:35:56.547 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:35:56.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:35:56.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:35:56.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.935154346999994, 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.557 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.441131526999996, 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:35:56.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:35:56.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:35:56.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:35:56.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:35:56.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.432122692999997, 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.563 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:35:56.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:35:56.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.930423744999999, 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:35:56.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:35:56.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.930361398999999, 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:35:56.571 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:35:56.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:35:56.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.929851720000002, 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.575 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:35:56.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.929740658, 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.579 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:35:56.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:35:56.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:35:56.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.929675392, 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.583 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:35:56.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:56.587 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.929615325999997, 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:35:56.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:35:56.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.591 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.929558632999999, 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:56.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.595 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.9294893, 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:35:56.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:35:56.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:35:56.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.929398681000002, 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:35:56.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:35:56.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.929355268000002, 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:35:56.604 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:35:56.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:35:56.608 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:35:56.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:35:56.611 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:56.604+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:56.635 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:35:56.636 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:35:56.636 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-24T00:35:56.636 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:35:56.636 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:35:56.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-24T00:35:56.711 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:56.707+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:56.755 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:56.755 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:56.755 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:56.755 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:56.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:35:56.784 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:56.784 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "" 2024-04-24T00:35:56.785 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:56.845 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:56.840+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:56.889 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:56.889 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:56.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:35:56.913 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:56.907+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:57.013 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:57.007+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:57.075 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-04-24T00:35:57.075 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:57.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:35:57.114 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:35:57.111+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:35:57.215 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:57.215 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:57.215 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:57.215 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:57.215 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:57.211+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:57.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:35:57.316 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:57.308+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:57.354 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:57.355 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:57.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:35:57.454 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:57.448+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:57.519 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:57.512+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.103347142000004, 2024-04-24T00:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.047835656000004, 2024-04-24T00:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.947259008000003, 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.905694053999994, 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.341962492, 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.838989094999999, 2024-04-24T00:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.600674861999998, 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565734439, 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565713565999999, 2024-04-24T00:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565690187000001, 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565668644999999, 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565647125000002, 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565600967999998, 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565576848999999, 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565553444999999, 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565525672, 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565504259000001, 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.565483667999999, 2024-04-24T00:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.349413426000002, 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.321442522000002, 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:57.640 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:57.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:35:57.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:35:57.642 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:57.636+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:57.743 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:57.736+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:57.814 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:57.814 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:57.814 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:35:57.814 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:57.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:35:57.874 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:57.868+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:57.899 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-04-24T00:35:57.945 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:57.936+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:57.999 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:35:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:35:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:35:57.999 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:58.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:35:58.059 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:58.052+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:58.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:35:58.147 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:58.143+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:58.167 INFO:teuthology.orchestra.run.smithi064.stdout:3542 write 0x51376dd thru 0x5142e0c (0xb730 bytes) 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 91.595170689, 2024-04-24T00:35:58.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:35:58.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 91.595096834000003, 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.187 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:35:58.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.898402050000001, 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:35:58.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:35:58.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 71.092870090999995, 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.196 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:35:58.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:77813 3.0 e102/21 3:05afdb1e:::rbd_data.115e805f80b3.0000000000000004:head v 102'237, mlcod=102'235)", 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.513429742, 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074761180000000003, 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664825+0000", 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664911+0000", 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5873999999999998e-05 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664944+0000", 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664959+0000", 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5979000000000001e-05 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665112+0000", 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152943 2024-04-24T00:35:58.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665157+0000", 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4725000000000003e-05 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672178+0000", 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070206390000000004 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672298+0000", 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120308 2024-04-24T00:35:58.204 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head, version: 45'133, data_included: [0~131072,196158~37038,262144~131072,524288~131072,786432~262144,1258308~52412,1441792~131072,3014656~131072,3267061~6628,6160384~34994], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head@45'133, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 6195378, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.256190659000001, 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011957290000000001, 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922063+0000", 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.306e-06 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923213+0000", 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011495310000000001 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923258+0000", 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4747999999999999e-05 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923284+0000", 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5873000000000001e-05 2024-04-24T00:35:58.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923436+0000", 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015226500000000001 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923452+0000", 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6022e-05 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.934018+0000", 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010566545 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head, version: 45'137, data_included: [0~131072,262144~131072,524288~131072,786432~199751,1179648~1552,1441792~3851,3014656~131072,5775043~64169,6160384~131072], data_size: 924683, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head@45'137, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8781824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.227365936999998, 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011581296, 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:58.209 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950888+0000", 2024-04-24T00:35:58.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951161+0000", 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027280499999999999 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951196+0000", 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123999999999998e-05 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951224+0000", 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8045e-05 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951356+0000", 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013187799999999999 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951376+0000", 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.003e-05 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.962467+0000", 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011091474 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.022031122000001, 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:35:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head, version: 80'319, data_included: [0~393216,524288~131072,1179648~524288], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head@80'319, size: 13631488, copy_subset: [(0, 13631488)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:35:58.217 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.009277029, 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089307889999999997, 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168976+0000", 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169620+0000", 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064358299999999998 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169650+0000", 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0111e-05 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169671+0000", 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1231000000000001e-05 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169789+0000", 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118368 2024-04-24T00:35:58.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169803+0000", 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3779e-05 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.177905+0000", 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081024800000000004 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.997130082999998, 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:35:58.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.980385925, 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:35:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:35:58.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.890236932000001, 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:35:58.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:35:58.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89880 3.0 3:05091b21:::rbd_data.115e6713a289.0000000000000001:30 [sparse-read 1310720~262144 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.638249+0000", 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5400026400000009, 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066749640000000002, 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:58.235 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89880 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638249+0000", 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638249+0000", 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638250+0000", 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4600000000000003e-07 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638253+0000", 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2590000000000001e-06 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638262+0000", 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8540000000000003e-06 2024-04-24T00:35:58.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.639337+0000", 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001075194 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.643999+0000", 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00466132 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.644092+0000", 2024-04-24T00:35:58.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3208999999999995e-05 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.644924+0000", 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083228200000000005 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89926 3.12 3:49ee3472:::rbd_object_map.115e56aa02fc.0000000000000031:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.494779608, 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068252479999999999, 2024-04-24T00:35:58.239 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89926 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683473+0000", 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8899999999999998e-07 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683491+0000", 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7829000000000002e-05 2024-04-24T00:35:58.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683506+0000", 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5457000000000001e-05 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683520+0000", 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3186e-05 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.684785+0000", 2024-04-24T00:35:58.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001265544 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.684895+0000", 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010945 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.685272+0000", 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037777200000000001 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.687975+0000", 2024-04-24T00:35:58.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027025759999999999 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690177+0000", 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002202196 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690241+0000", 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4312000000000004e-05 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690297+0000", 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5937000000000003e-05 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89927 3.12 3:49ee3472:::rbd_object_map.115e56aa02fc.0000000000000031:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4916878400000009, 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068456009999999998, 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89927 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:35:58.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686566+0000", 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2079999999999999e-06 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686575+0000", 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8359999999999998e-06 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686622+0000", 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7533e-05 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686633+0000", 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0546e-05 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686726+0000", 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3431999999999996e-05 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686834+0000", 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010771 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.687193+0000", 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035882399999999999 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.689501+0000", 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023082760000000002 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693281+0000", 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037798509999999999 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693348+0000", 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6913000000000001e-05 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693409+0000", 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1471999999999997e-05 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4830802530000007, 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:35:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:35:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4726935559999994, 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:35:58.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:35:58.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.463750267, 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.261 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:90104 3.18 3:193b8512:::rbd_data.115e56aa02fc.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write 1644318~44706] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.0676173740000001, 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084175689999999997, 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 90104 2024-04-24T00:35:58.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110636+0000", 2024-04-24T00:35:58.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110706+0000", 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9392000000000002e-05 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110734+0000", 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8538999999999999e-05 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110747+0000", 2024-04-24T00:35:58.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860000000000001e-05 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110850+0000", 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102988 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111107+0000", 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025716600000000002 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111346+0000", 2024-04-24T00:35:58.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000238373 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.117684+0000", 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063379949999999999 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118724+0000", 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001040685 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118914+0000", 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018984699999999999 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.119052+0000", 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137701 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPushReply(3.12 128/127 [PushReplyOp(3:4e045fb2:::rbd_data.115e6f07caf6.0000000000000007:head)])", 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:35:58.271 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.8004280049999997, 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089128709999999993, 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:35:58.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377827+0000", 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5870000000000001e-06 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377852+0000", 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5083e-05 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377890+0000", 2024-04-24T00:35:58.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7279e-05 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377914+0000", 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4496000000000001e-05 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.378154+0000", 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024028 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:35:58.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:35:58.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.386737+0000", 2024-04-24T00:35:58.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085821460000000006 2024-04-24T00:35:58.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:35:58.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:35:58.275 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:35:58.276 INFO:teuthology.orchestra.run.smithi064.stdout:3543 trunc from 0xe8d695e to 0xa691640 2024-04-24T00:35:58.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:35:58.278 INFO:teuthology.orchestra.run.smithi064.stdout:3544 write 0x23e9d8c thru 0x23f2cce (0x8f43 bytes) 2024-04-24T00:35:58.278 INFO:teuthology.orchestra.run.smithi064.stdout:3545 read 0x411add4 thru 0x412165e (0x688b bytes) 2024-04-24T00:35:58.278 INFO:teuthology.orchestra.run.smithi064.stdout:3546 write 0x8682e0 thru 0x86d39a (0x50bb bytes) 2024-04-24T00:35:58.278 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:35:58.278 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:35:58.279 INFO:teuthology.orchestra.run.smithi064.stdout:3549 write 0xadd4506 thru 0xaddf95a (0xb455 bytes) 2024-04-24T00:35:58.279 INFO:teuthology.orchestra.run.smithi064.stdout:3550 write 0x7b15247 thru 0x7b15549 (0x303 bytes) 2024-04-24T00:35:58.279 INFO:teuthology.orchestra.run.smithi064.stdout:3551 write 0x63609a8 thru 0x6367d03 (0x735c bytes) 2024-04-24T00:35:58.279 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:35:58.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:35:58.280 INFO:teuthology.orchestra.run.smithi064.stdout:3552 trunc from 0xaddf95b to 0x7725361 2024-04-24T00:35:58.280 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:35:58.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:35:58.281 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:58.268+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:58.287 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:35:58.299 INFO:teuthology.orchestra.run.smithi064.stdout:3553 punch from 0x2b5f84 to 0x2be4f2, (0x856e bytes) 2024-04-24T00:35:58.299 INFO:teuthology.orchestra.run.smithi064.stdout:3554 read 0x6ed49d7 thru 0x6ed4fe7 (0x611 bytes) 2024-04-24T00:35:58.299 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:58.299 INFO:teuthology.orchestra.run.smithi064.stdout:3556 read 0x34dc6f5 thru 0x34eb0f2 (0xe9fe bytes) 2024-04-24T00:35:58.300 INFO:teuthology.orchestra.run.smithi064.stdout:3558 read 0xbdfc3c thru 0xbe9d49 (0xa10e bytes) 2024-04-24T00:35:58.301 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:58.301 INFO:teuthology.orchestra.run.smithi064.stdout:3561 punch from 0x501a193 to 0x5023cf0, (0x9b5d bytes) 2024-04-24T00:35:58.301 INFO:teuthology.orchestra.run.smithi064.stdout:3562 read 0x3cfdbe thru 0x3dd31a (0xd55d bytes) 2024-04-24T00:35:58.301 INFO:teuthology.orchestra.run.smithi064.stdout:3563 read 0x469c523 thru 0x46a16a3 (0x5181 bytes) 2024-04-24T00:35:58.310 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:35:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:35:58.323 INFO:teuthology.orchestra.run.smithi064.stdout:3565 write 0xd9fe0de thru 0xda012b0 (0x31d3 bytes) 2024-04-24T00:35:58.332 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:35:58.332 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:35:58.333 INFO:teuthology.orchestra.run.smithi064.stdout:3566 read 0x1759f34 thru 0x1761aab (0x7b78 bytes) 2024-04-24T00:35:58.333 INFO:teuthology.orchestra.run.smithi064.stdout:3567 read 0xbff0cb0 thru 0xbffd5fa (0xc94b bytes) 2024-04-24T00:35:58.333 INFO:teuthology.orchestra.run.smithi064.stdout:3568 writesame 0xd9686d4 thru 0xd96c4c2 data_size 0x5(0x3def bytes) 2024-04-24T00:35:58.353 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:35:58.354 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:35:58.355 INFO:teuthology.orchestra.run.smithi064.stdout:3569 trunc from 0xda012b1 to 0x9ecd77e 2024-04-24T00:35:58.377 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:35:58.377 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:35:58.378 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:58.375+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:58.378 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:58.378 INFO:teuthology.orchestra.run.smithi064.stdout:3573 read 0x79a4022 thru 0x79b0582 (0xc561 bytes) 2024-04-24T00:35:58.378 INFO:teuthology.orchestra.run.smithi064.stdout:3574 read 0x570f4a4 thru 0x5710b38 (0x1695 bytes) 2024-04-24T00:35:58.379 INFO:teuthology.orchestra.run.smithi064.stdout:3576 write 0x9b371ee thru 0x9b43c4b (0xca5e bytes) 2024-04-24T00:35:58.401 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:58.401 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:58.404 INFO:teuthology.orchestra.run.smithi064.stdout:3578 writesame 0xc8a4f8f thru 0xc8b437a data_size 0x86(0xf3ec bytes) 2024-04-24T00:35:58.404 INFO:teuthology.orchestra.run.smithi064.stdout:3580 trunc from 0xc8b437b to 0xc53e855 2024-04-24T00:35:58.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:35:58.416 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:58.418 INFO:teuthology.orchestra.run.smithi064.stdout:3583 trunc from 0xc53e855 to 0xaa1b2e0 2024-04-24T00:35:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:3585 read 0x8edc77 thru 0x8f0aa9 (0x2e33 bytes) 2024-04-24T00:35:58.434 INFO:teuthology.orchestra.run.smithi064.stdout:3586 trunc from 0xaa1b2e0 to 0xce196a3 2024-04-24T00:35:58.446 INFO:teuthology.orchestra.run.smithi064.stdout:3588 write 0x3972c3d thru 0x39743e2 (0x17a6 bytes) 2024-04-24T00:35:58.451 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:58.451 INFO:teuthology.orchestra.run.smithi064.stdout:3590 punch from 0x11f20be to 0x120199d, (0xf8df bytes) 2024-04-24T00:35:58.451 INFO:teuthology.orchestra.run.smithi064.stdout:3591 read 0x118ebc0 thru 0x119ae2a (0xc26b bytes) 2024-04-24T00:35:58.451 INFO:teuthology.orchestra.run.smithi064.stdout:3592 write 0x520737 thru 0x52abec (0xa4b6 bytes) 2024-04-24T00:35:58.475 INFO:teuthology.orchestra.run.smithi064.stdout:3594 trunc from 0xce196a3 to 0xccb74e 2024-04-24T00:35:58.478 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:58.472+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:58.526 INFO:teuthology.orchestra.run.smithi064.stdout:3595 write 0x474cc8d thru 0x4759443 (0xc7b7 bytes) 2024-04-24T00:35:58.531 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:58.532 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:58.544 INFO:teuthology.orchestra.run.smithi064.stdout:3596 trunc from 0x4759444 to 0x9cf8c86 2024-04-24T00:35:58.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:35:58.576 INFO:teuthology.orchestra.run.smithi064.stdout:3597 write 0xa13a49f thru 0xa13f3ad (0x4f0f bytes) 2024-04-24T00:35:58.583 INFO:teuthology.orchestra.run.smithi064.stdout:3599 read 0x9c07fdf thru 0x9c0c48a (0x44ac bytes) 2024-04-24T00:35:58.583 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:58.584 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:58.576+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:58.587 INFO:teuthology.orchestra.run.smithi064.stdout:3603 trunc from 0xa13f3ae to 0x8083b93 2024-04-24T00:35:58.624 INFO:teuthology.orchestra.run.smithi064.stdout:3604 write 0xa69e74d thru 0xa6a2912 (0x41c6 bytes) 2024-04-24T00:35:58.633 INFO:teuthology.orchestra.run.smithi064.stdout:3606 write 0x6320418 thru 0x63298a3 (0x948c bytes) 2024-04-24T00:35:58.643 INFO:teuthology.orchestra.run.smithi064.stdout:3608 read 0x6b279a7 thru 0x6b34d38 (0xd392 bytes) 2024-04-24T00:35:58.644 INFO:teuthology.orchestra.run.smithi064.stdout:3612 read 0x3cf691 thru 0x3d5560 (0x5ed0 bytes) 2024-04-24T00:35:58.645 INFO:teuthology.orchestra.run.smithi064.stdout:3613 read 0x45e9bf8 thru 0x45f8851 (0xec5a bytes) 2024-04-24T00:35:58.645 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:58.646 INFO:teuthology.orchestra.run.smithi064.stdout:3617 write 0x7ecc496 thru 0x7ed01cb (0x3d36 bytes) 2024-04-24T00:35:58.653 INFO:teuthology.orchestra.run.smithi064.stdout:3618 write 0x65454a0 thru 0x654cde0 (0x7941 bytes) 2024-04-24T00:35:58.659 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:35:58.682 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:58.675+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:58.683 INFO:teuthology.orchestra.run.smithi064.stdout:3619 writesame 0xac214b6 thru 0xac21cbe data_size 0x11(0x809 bytes) 2024-04-24T00:35:58.699 INFO:teuthology.orchestra.run.smithi064.stdout:3621 writesame 0xe46d966 thru 0xe479fe6 data_size 0xd(0xc681 bytes) 2024-04-24T00:35:58.703 INFO:teuthology.orchestra.run.smithi064.stdout:3622 trunc from 0xe479fe7 to 0xb29faed 2024-04-24T00:35:58.711 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:58.712 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:58.717 INFO:teuthology.orchestra.run.smithi064.stdout:3623 writesame 0x1e0be2b thru 0x1e1ba76 data_size 0x4(0xfc4c bytes) 2024-04-24T00:35:58.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:35:58.767 INFO:teuthology.orchestra.run.smithi064.stdout:3624 read 0x6ded289 thru 0x6df5719 (0x8491 bytes) 2024-04-24T00:35:58.767 INFO:teuthology.orchestra.run.smithi064.stdout:3626 read 0x52e729d thru 0x52eb36a (0x40ce bytes) 2024-04-24T00:35:58.767 INFO:teuthology.orchestra.run.smithi064.stdout:3627 write 0x9025149 thru 0x9031273 (0xc12b bytes) 2024-04-24T00:35:58.776 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:58.776 INFO:teuthology.orchestra.run.smithi064.stdout:3632 read 0x93c9b08 thru 0x93d2287 (0x8780 bytes) 2024-04-24T00:35:58.779 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:58.775+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:58.792 INFO:teuthology.orchestra.run.smithi064.stdout:3633 trunc from 0xb29faed to 0x22c383 2024-04-24T00:35:58.838 INFO:teuthology.orchestra.run.smithi064.stdout:3634 write 0x33ab5cf thru 0x33b1992 (0x63c4 bytes) 2024-04-24T00:35:58.843 INFO:teuthology.orchestra.run.smithi064.stdout:3635 read 0x2eef473 thru 0x2ef7777 (0x8305 bytes) 2024-04-24T00:35:58.864 INFO:teuthology.orchestra.run.smithi064.stdout:3636 write 0x79459a7 thru 0x7948e74 (0x34ce bytes) 2024-04-24T00:35:58.864 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:58.865 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:58.871 INFO:teuthology.orchestra.run.smithi064.stdout:3637 punch from 0x6fbe246 to 0x6fbfd71, (0x1b2b bytes) 2024-04-24T00:35:58.871 INFO:teuthology.orchestra.run.smithi064.stdout:3638 read 0x1135980 thru 0x113a64d (0x4cce bytes) 2024-04-24T00:35:58.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:35:58.882 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:58.876+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:58.882 INFO:teuthology.orchestra.run.smithi064.stdout:3639 trunc from 0x7948e75 to 0x2926d7c 2024-04-24T00:35:58.904 INFO:teuthology.orchestra.run.smithi064.stdout:3640 punch from 0x21832a7 to 0x2185ddd, (0x2b36 bytes) 2024-04-24T00:35:58.908 INFO:teuthology.orchestra.run.smithi064.stdout:3641 trunc from 0x2926d7c to 0x45db9c7 2024-04-24T00:35:58.922 INFO:teuthology.orchestra.run.smithi064.stdout:3642 read 0x4de230 thru 0x4e47c3 (0x6594 bytes) 2024-04-24T00:35:58.957 INFO:teuthology.orchestra.run.smithi064.stdout:3644 write 0xd947c3c thru 0xd94ef02 (0x72c7 bytes) 2024-04-24T00:35:58.963 INFO:teuthology.orchestra.run.smithi064.stdout:3645 write 0x629f509 thru 0x62a5277 (0x5d6f bytes) 2024-04-24T00:35:58.970 INFO:teuthology.orchestra.run.smithi064.stdout:3646 read 0x2963ad0 thru 0x296bac5 (0x7ff6 bytes) 2024-04-24T00:35:58.974 INFO:teuthology.orchestra.run.smithi064.stdout:3647 trunc from 0xd94ef03 to 0xec63f6d 2024-04-24T00:35:58.982 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:35:58.976+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:35:59.018 INFO:teuthology.orchestra.run.smithi064.stdout:3648 trunc from 0xec63f6d to 0x27064f4 2024-04-24T00:35:59.048 INFO:teuthology.orchestra.run.smithi064.stdout:3649 read 0x2629108 thru 0x263185d (0x8756 bytes) 2024-04-24T00:35:59.050 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:59.050 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:59.055 INFO:teuthology.orchestra.run.smithi064.stdout:3650 trunc from 0x27064f4 to 0x4e472f0 2024-04-24T00:35:59.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:35:59.096 INFO:teuthology.orchestra.run.smithi064.stdout:3651 writesame 0xb42de5a thru 0xb432f1f data_size 0xe(0x50c6 bytes) 2024-04-24T00:35:59.096 INFO:teuthology.orchestra.run.smithi064.stdout:3652 read 0x7a102e8 thru 0x7a18246 (0x7f5f bytes) 2024-04-24T00:35:59.096 INFO:teuthology.orchestra.run.smithi064.stdout:3654 writesame 0x7aff8c0 thru 0x7b0a701 data_size 0xf(0xae42 bytes) 2024-04-24T00:35:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:3655 write 0x19e8177 thru 0x19f5633 (0xd4bd bytes) 2024-04-24T00:35:59.110 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:35:59.107+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:35:59.120 INFO:teuthology.orchestra.run.smithi064.stdout:3656 write 0xb18dd46 thru 0xb19a764 (0xca1f bytes) 2024-04-24T00:35:59.132 INFO:teuthology.orchestra.run.smithi064.stdout:3657 read 0x927ea2a thru 0x928d86d (0xee44 bytes) 2024-04-24T00:35:59.132 INFO:teuthology.orchestra.run.smithi064.stdout:3658 read 0x57483b0 thru 0x5756dfc (0xea4d bytes) 2024-04-24T00:35:59.141 INFO:teuthology.orchestra.run.smithi064.stdout:3659 writesame 0xb8249ec thru 0xb825e2d data_size 0x1(0x1442 bytes) 2024-04-24T00:35:59.153 INFO:teuthology.orchestra.run.smithi064.stdout:3660 trunc from 0xb825e2e to 0xebc2ce1 2024-04-24T00:35:59.183 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:59.179+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:59.190 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:59.190 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:59.195 INFO:teuthology.orchestra.run.smithi064.stdout:3663 trunc from 0xebc2ce1 to 0x348fb8f 2024-04-24T00:35:59.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:35:59.237 INFO:teuthology.orchestra.run.smithi064.stdout:3664 write 0x745ffb2 thru 0x746f337 (0xf386 bytes) 2024-04-24T00:35:59.244 INFO:teuthology.orchestra.run.smithi064.stdout:3666 write 0x2b0662 thru 0x2b9095 (0x8a34 bytes) 2024-04-24T00:35:59.265 INFO:teuthology.orchestra.run.smithi064.stdout:3668 write 0x817d428 thru 0x818caf3 (0xf6cc bytes) 2024-04-24T00:35:59.271 INFO:teuthology.orchestra.run.smithi064.stdout:3669 read 0x4ea123a thru 0x4eb073f (0xf506 bytes) 2024-04-24T00:35:59.272 INFO:teuthology.orchestra.run.smithi064.stdout:3670 punch from 0x772c200 to 0x773641a, (0xa21a bytes) 2024-04-24T00:35:59.272 INFO:teuthology.orchestra.run.smithi064.stdout:3672 punch from 0x7f7bd68 to 0x7f7fdc9, (0x4061 bytes) 2024-04-24T00:35:59.272 INFO:teuthology.orchestra.run.smithi064.stdout:3674 read 0x89c934 thru 0x8a203a (0x5707 bytes) 2024-04-24T00:35:59.272 INFO:teuthology.orchestra.run.smithi064.stdout:3675 read 0x7649873 thru 0x764bf26 (0x26b4 bytes) 2024-04-24T00:35:59.282 INFO:teuthology.orchestra.run.smithi064.stdout:3677 trunc from 0x818caf4 to 0xb57f280 2024-04-24T00:35:59.284 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:59.276+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:35:59.294 INFO:teuthology.orchestra.run.smithi064.stdout:3678 punch from 0xa9f8471 to 0xa9fc968, (0x44f7 bytes) 2024-04-24T00:35:59.315 INFO:teuthology.orchestra.run.smithi064.stdout:3679 writesame 0xeb2fdc7 thru 0xeb30c77 data_size 0x1(0xeb1 bytes) 2024-04-24T00:35:59.315 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:59.315 INFO:teuthology.orchestra.run.smithi064.stdout:3683 read 0x7aaf493 thru 0x7ab4f10 (0x5a7e bytes) 2024-04-24T00:35:59.315 INFO:teuthology.orchestra.run.smithi064.stdout:3685 read 0xb3b497c thru 0xb3be9b4 (0xa039 bytes) 2024-04-24T00:35:59.315 INFO:teuthology.orchestra.run.smithi064.stdout:3686 writesame 0xa3dded9 thru 0xa3e646a data_size 0x52(0x8592 bytes) 2024-04-24T00:35:59.322 INFO:teuthology.orchestra.run.smithi064.stdout:3687 read 0xb86e914 thru 0xb875804 (0x6ef1 bytes) 2024-04-24T00:35:59.322 INFO:teuthology.orchestra.run.smithi064.stdout:3689 write 0xbe67150 thru 0xbe69152 (0x2003 bytes) 2024-04-24T00:35:59.329 INFO:teuthology.orchestra.run.smithi064.stdout:3690 punch from 0x6188c05 to 0x618f247, (0x6642 bytes) 2024-04-24T00:35:59.329 INFO:teuthology.orchestra.run.smithi064.stdout:3692 write 0xbc80f0f thru 0xbc8bbe2 (0xacd4 bytes) 2024-04-24T00:35:59.340 INFO:teuthology.orchestra.run.smithi064.stdout:3694 writesame 0x14837e9 thru 0x149347e data_size 0x27(0xfc96 bytes) 2024-04-24T00:35:59.346 INFO:teuthology.orchestra.run.smithi064.stdout:3695 write 0x4882c7e thru 0x4888483 (0x5806 bytes) 2024-04-24T00:35:59.355 INFO:teuthology.orchestra.run.smithi064.stdout:3697 punch from 0x82181c6 to 0x822202d, (0x9e67 bytes) 2024-04-24T00:35:59.355 INFO:teuthology.orchestra.run.smithi064.stdout:3698 write 0xe2353c8 thru 0xe23c656 (0x728f bytes) 2024-04-24T00:35:59.366 INFO:teuthology.orchestra.run.smithi064.stdout:3699 read 0x313c07f thru 0x314272d (0x66af bytes) 2024-04-24T00:35:59.366 INFO:teuthology.orchestra.run.smithi064.stdout:3700 writesame 0xf863c8 thru 0xf93e73 data_size 0x5a(0xdaac bytes) 2024-04-24T00:35:59.385 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:35:59.383+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:35:59.386 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:59.386 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:59.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:35:59.403 INFO:teuthology.orchestra.run.smithi064.stdout:3701 trunc from 0xeb30c78 to 0x2fcd66f 2024-04-24T00:35:59.427 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:59.427 INFO:teuthology.orchestra.run.smithi064.stdout:3704 punch from 0x1879c90 to 0x18828a5, (0x8c15 bytes) 2024-04-24T00:35:59.427 INFO:teuthology.orchestra.run.smithi064.stdout:3705 read 0x13aec85 thru 0x13b80b1 (0x942d bytes) 2024-04-24T00:35:59.427 INFO:teuthology.orchestra.run.smithi064.stdout:3706 punch from 0x18975ae to 0x1898ccd, (0x171f bytes) 2024-04-24T00:35:59.432 INFO:teuthology.orchestra.run.smithi064.stdout:3707 trunc from 0x2fcd66f to 0x1069b1f 2024-04-24T00:35:59.447 INFO:teuthology.orchestra.run.smithi064.stdout:3708 read 0x6ebaa9 thru 0x6f277f (0x6cd7 bytes) 2024-04-24T00:35:59.448 INFO:teuthology.orchestra.run.smithi064.stdout:3709 punch from 0x4e7387 to 0x4e8610, (0x1289 bytes) 2024-04-24T00:35:59.486 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:35:59.480+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:35:59.488 INFO:teuthology.orchestra.run.smithi064.stdout:3710 writesame 0xc3b9447 thru 0xc3c365f data_size 0x1(0xa219 bytes) 2024-04-24T00:35:59.489 INFO:teuthology.orchestra.run.smithi064.stdout:3711 read 0x35ad910 thru 0x35ba780 (0xce71 bytes) 2024-04-24T00:35:59.489 INFO:teuthology.orchestra.run.smithi064.stdout:3713 read 0x777d8c8 thru 0x77850e5 (0x781e bytes) 2024-04-24T00:35:59.489 INFO:teuthology.orchestra.run.smithi064.stdout:3714 writesame 0x7baf95a thru 0x7bb7051 data_size 0x51(0x76f8 bytes) 2024-04-24T00:35:59.497 INFO:teuthology.orchestra.run.smithi064.stdout:3716 trunc from 0xc3c3660 to 0xd929374 2024-04-24T00:35:59.507 INFO:teuthology.orchestra.run.smithi064.stdout:3717 writesame 0x343829e thru 0x343e802 data_size 0x1(0x6565 bytes) 2024-04-24T00:35:59.570 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:59.570 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:59.570 INFO:teuthology.orchestra.run.smithi064.stdout:3721 read 0x7002c2b thru 0x701200d (0xf3e3 bytes) 2024-04-24T00:35:59.570 INFO:teuthology.orchestra.run.smithi064.stdout:3723 read 0x17edc5b thru 0x17fd622 (0xf9c8 bytes) 2024-04-24T00:35:59.570 INFO:teuthology.orchestra.run.smithi064.stdout:3724 write 0x1653266 thru 0x165e5e8 (0xb383 bytes) 2024-04-24T00:35:59.577 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:59.577 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:59.581 INFO:teuthology.orchestra.run.smithi064.stdout:3725 read 0xbd738ed thru 0xbd7b65d (0x7d71 bytes) 2024-04-24T00:35:59.582 INFO:teuthology.orchestra.run.smithi064.stdout:3726 read 0x8eb17bd thru 0x8eba136 (0x897a bytes) 2024-04-24T00:35:59.587 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:59.580+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:59.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:35:59.596 INFO:teuthology.orchestra.run.smithi064.stdout:3727 write 0xe8f5d09 thru 0xe904c24 (0xef1c bytes) 2024-04-24T00:35:59.601 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:59.601 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:35:59.601 INFO:teuthology.orchestra.run.smithi064.stdout:3731 clone 25 order 24 su 1048576 sc 12 2024-04-24T00:35:59.688 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:35:59.680+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:35:59.717 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:59.717 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:59.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:35:59.788 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:59.783+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:59.859 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:35:59.859 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:35:59.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:35:59.919 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:35:59.915+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:35:59.991 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:35:59.984+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:00.042 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:00.042 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:00.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:00.092 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:00.088+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:00.179 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:00.179 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:00.193 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:00.187+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:00.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:00.294 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:00.288+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:00.361 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:00.362 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:00.376 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:00.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:00.395 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:00.392+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:00.495 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:00.491+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:00.498 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:00.498 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:00.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:00.553 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:00.554 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:00.570 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:00.570 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:00.589 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:00.590 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:00.596 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:00.588+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:00.610 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:00.610 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:00.629 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:00.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:00.649 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:00.649 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:00.674 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:00.674 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:00.675 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:00.675 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:00.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:36:00.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:00.697 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:00.691+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:00.699 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:00.700 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:00.798 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:00.795+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:00.829 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:00.829 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:00.829 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:00.829 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:00.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:00.899 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:00.892+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:00.962 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:00.962 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:00.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:01.000 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:00.995+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:01.102 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:01.096+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:01.133 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:01.134 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:01.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:01.134 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-04-24T00:36:01.134 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:01.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:01.234 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:01.228+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:01.303 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:01.296+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:01.315 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:01.316 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:01.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:01.417 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:01.412+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:01.493 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:01.493 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:01.493 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:01.493 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:01.493 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:01.505 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:01.503+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:01.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:01.606 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:01.603+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:01.672 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:01.672 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:01.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.233850523000001, 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:01.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:36:01.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.178339037000001, 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.679 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.077762389, 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.683 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.688 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.036197435000005, 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:01.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.472465872999997, 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:36:01.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:36:01.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:36:01.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:01.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:01.699 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.969492475999999, 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:01.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:36:01.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.704 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.731178242999999, 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:36:01.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:01.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.69623782, 2024-04-24T00:36:01.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:36:01.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:01.713 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696216947, 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:01.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:36:01.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696193567999998, 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.718 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.722 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696172025999999, 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:36:01.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:36:01.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696150505999999, 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.727 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.731 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696104348999999, 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:36:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.69608023, 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.740 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696056826, 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:36:01.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:36:01.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696029053, 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:36:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:36:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696007640000001, 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:36:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:36:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:36:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:36:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.695987048999999, 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:36:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:36:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:36:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.479916806999999, 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:36:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:36:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:36:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.451945902999999, 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:36:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:36:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:36:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:36:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:01.769 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:01.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:01.772 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:36:01.772 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:01.764+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:01.800 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:01.800 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:01.801 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.9623898309624939 2024-04-24T00:36:01.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 0.9623898309624939 2024-04-24T00:36:01.872 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:01.867+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:01.887 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-04-24T00:36:01.952 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:01.953 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:01.972 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:01.964+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:01.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:02.073 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:02.068+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:02.134 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:02.134 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:02.134 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:02.134 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:02.135 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:02.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:02.190 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:02.184+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:02.275 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:02.268+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:02.300 INFO:teuthology.orchestra.run.smithi064.stdout:3732 writesame 0xb0cba00 thru 0xb0cbc53 data_size 0x4(0x254 bytes) 2024-04-24T00:36:02.308 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.720829335999994, 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:36:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:36:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.720755480999998, 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.024060696999996, 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:36:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:36:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.218528738000003, 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:36:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:36:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:77813 3.0 e102/21 3:05afdb1e:::rbd_data.115e805f80b3.0000000000000004:head v 102'237, mlcod=102'235)", 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.639088389000001, 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074761180000000003, 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664825+0000", 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664911+0000", 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5873999999999998e-05 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664944+0000", 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664959+0000", 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5979000000000001e-05 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665112+0000", 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152943 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665157+0000", 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4725000000000003e-05 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672178+0000", 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070206390000000004 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672298+0000", 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120308 2024-04-24T00:36:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head, version: 45'133, data_included: [0~131072,196158~37038,262144~131072,524288~131072,786432~262144,1258308~52412,1441792~131072,3014656~131072,3267061~6628,6160384~34994], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head@45'133, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 6195378, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.381849305999999, 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011957290000000001, 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922063+0000", 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.306e-06 2024-04-24T00:36:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923213+0000", 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011495310000000001 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923258+0000", 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4747999999999999e-05 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923284+0000", 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5873000000000001e-05 2024-04-24T00:36:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923436+0000", 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015226500000000001 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923452+0000", 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6022e-05 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.934018+0000", 2024-04-24T00:36:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010566545 2024-04-24T00:36:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head, version: 45'137, data_included: [0~131072,262144~131072,524288~131072,786432~199751,1179648~1552,1441792~3851,3014656~131072,5775043~64169,6160384~131072], data_size: 924683, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head@45'137, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8781824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.353024584, 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011581296, 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950888+0000", 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951161+0000", 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027280499999999999 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951196+0000", 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123999999999998e-05 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951224+0000", 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8045e-05 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951356+0000", 2024-04-24T00:36:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013187799999999999 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951376+0000", 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.003e-05 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.962467+0000", 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011091474 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.147689768999999, 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head, version: 80'319, data_included: [0~393216,524288~131072,1179648~524288], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head@80'319, size: 13631488, copy_subset: [(0, 13631488)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.134935676000001, 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089307889999999997, 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168976+0000", 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169620+0000", 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064358299999999998 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169650+0000", 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0111e-05 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169671+0000", 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1231000000000001e-05 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169789+0000", 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118368 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169803+0000", 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3779e-05 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.177905+0000", 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081024800000000004 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.12278873, 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.106044571999998, 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.015895578999999, 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89880 3.0 3:05091b21:::rbd_data.115e6713a289.0000000000000001:30 [sparse-read 1310720~262144 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.638249+0000", 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.665661287000001, 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066749640000000002, 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89880 2024-04-24T00:36:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638249+0000", 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638249+0000", 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638250+0000", 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4600000000000003e-07 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638253+0000", 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2590000000000001e-06 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.638262+0000", 2024-04-24T00:36:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8540000000000003e-06 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.639337+0000", 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001075194 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.643999+0000", 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00466132 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.644092+0000", 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3208999999999995e-05 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.644924+0000", 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083228200000000005 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89926 3.12 3:49ee3472:::rbd_object_map.115e56aa02fc.0000000000000031:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.620438255, 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068252479999999999, 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89926 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683473+0000", 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8899999999999998e-07 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683491+0000", 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7829000000000002e-05 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683506+0000", 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5457000000000001e-05 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683520+0000", 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3186e-05 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.684785+0000", 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001265544 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.684895+0000", 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010945 2024-04-24T00:36:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.685272+0000", 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037777200000000001 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.687975+0000", 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027025759999999999 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690177+0000", 2024-04-24T00:36:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002202196 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690241+0000", 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4312000000000004e-05 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690297+0000", 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5937000000000003e-05 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89927 3.12 3:49ee3472:::rbd_object_map.115e56aa02fc.0000000000000031:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.617346487000001, 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068456009999999998, 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89927 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686566+0000", 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2079999999999999e-06 2024-04-24T00:36:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686575+0000", 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8359999999999998e-06 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686622+0000", 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7533e-05 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686633+0000", 2024-04-24T00:36:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0546e-05 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686726+0000", 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3431999999999996e-05 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686834+0000", 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010771 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.687193+0000", 2024-04-24T00:36:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035882399999999999 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.689501+0000", 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023082760000000002 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693281+0000", 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037798509999999999 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693348+0000", 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6913000000000001e-05 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693409+0000", 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1471999999999997e-05 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.608738900000001, 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.598352202999999, 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.589408914, 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:90104 3.18 3:193b8512:::rbd_data.115e56aa02fc.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write 1644318~44706] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.193276021000001, 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084175689999999997, 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 90104 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110636+0000", 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110706+0000", 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9392000000000002e-05 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110734+0000", 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8538999999999999e-05 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110747+0000", 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860000000000001e-05 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110850+0000", 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102988 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111107+0000", 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025716600000000002 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111346+0000", 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000238373 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.117684+0000", 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063379949999999999 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118724+0000", 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001040685 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118914+0000", 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018984699999999999 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.119052+0000", 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137701 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPushReply(3.12 128/127 [PushReplyOp(3:4e045fb2:::rbd_data.115e6f07caf6.0000000000000007:head)])", 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9260866520000004, 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089128709999999993, 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377827+0000", 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5870000000000001e-06 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377852+0000", 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5083e-05 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377890+0000", 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7279e-05 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377914+0000", 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4496000000000001e-05 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.378154+0000", 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024028 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.386737+0000", 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085821460000000006 2024-04-24T00:36:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.408 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:02.410 INFO:teuthology.orchestra.run.smithi064.stdout:3733 read 0x9c6ff94 thru 0x9c74cd9 (0x4d46 bytes) 2024-04-24T00:36:02.410 INFO:teuthology.orchestra.run.smithi064.stdout:3734 read 0xb94f7cf thru 0xb957d0f (0x8541 bytes) 2024-04-24T00:36:02.410 INFO:teuthology.orchestra.run.smithi064.stdout:3735 read 0xb20a619 thru 0xb21460c (0x9ff4 bytes) 2024-04-24T00:36:02.410 INFO:teuthology.orchestra.run.smithi064.stdout:3736 writesame 0x31bc639 thru 0x31bdb1d data_size 0x1(0x14e5 bytes) 2024-04-24T00:36:02.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:3737 writesame 0x223d0 thru 0x22a15 data_size 0x2(0x646 bytes) 2024-04-24T00:36:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:3738 punch from 0x1b02129 to 0x1b03d30, (0x1c07 bytes) 2024-04-24T00:36:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:3739 read 0xab12c7c thru 0xab1d3b0 (0xa735 bytes) 2024-04-24T00:36:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:3740 read 0x8cbc1e1 thru 0x8cc20c6 (0x5ee6 bytes) 2024-04-24T00:36:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:3741 write 0x5f503b9 thru 0x5f5b1cc (0xae14 bytes) 2024-04-24T00:36:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:3742 write 0x536abfe thru 0x5370463 (0x5866 bytes) 2024-04-24T00:36:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:3745 write 0x354f011 thru 0x355b811 (0xc801 bytes) 2024-04-24T00:36:02.413 INFO:teuthology.orchestra.run.smithi064.stdout:3746 writesame 0x47c09f3 thru 0x47cbb89 data_size 0x1(0xb197 bytes) 2024-04-24T00:36:02.413 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:02.404+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:02.472 INFO:teuthology.orchestra.run.smithi064.stdout:3749 write 0x84b62f8 thru 0x84bcf7e (0x6c87 bytes) 2024-04-24T00:36:02.477 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:02.478 INFO:teuthology.orchestra.run.smithi064.stdout:3751 read 0x51d5bfb thru 0x51da581 (0x4987 bytes) 2024-04-24T00:36:02.478 INFO:teuthology.orchestra.run.smithi064.stdout:3752 punch from 0x43b6456 to 0x43c40c8, (0xdc72 bytes) 2024-04-24T00:36:02.485 INFO:teuthology.orchestra.run.smithi064.stdout:3753 trunc from 0xe904c25 to 0x9eb4209 2024-04-24T00:36:02.511 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:02.504+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:02.523 INFO:teuthology.orchestra.run.smithi064.stdout:3754 read 0x81e2dc2 thru 0x81e34f7 (0x736 bytes) 2024-04-24T00:36:02.523 INFO:teuthology.orchestra.run.smithi064.stdout:3755 read 0x4584596 thru 0x4588e1b (0x4886 bytes) 2024-04-24T00:36:02.529 INFO:teuthology.orchestra.run.smithi064.stdout:3758 trunc from 0x9eb4209 to 0xe125dc6 2024-04-24T00:36:02.542 INFO:teuthology.orchestra.run.smithi064.stdout:3759 read 0x1d72586 thru 0x1d7e2ac (0xbd27 bytes) 2024-04-24T00:36:02.543 INFO:teuthology.orchestra.run.smithi064.stdout:3760 writesame 0x2f962c0 thru 0x2f9e68a data_size 0x1(0x83cb bytes) 2024-04-24T00:36:02.578 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.034720045999999997, 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034751637000000002, 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91840 3.15 e138/106)", 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.565311+0000", 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.012048006, 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012127092000000001, 2024-04-24T00:36:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.565311+0000", 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.565311+0000", 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.565313+0000", 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099999999999999e-06 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.565320+0000", 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3359999999999997e-06 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.565345+0000", 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4847000000000001e-05 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.565359+0000", 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4263e-05 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 2 2024-04-24T00:36:02.586 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:02.590 INFO:teuthology.orchestra.run.smithi064.stdout:3761 write 0x44f476a thru 0x44fd80d (0x90a4 bytes) 2024-04-24T00:36:02.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:02.598 INFO:teuthology.orchestra.run.smithi064.stdout:3762 read 0x8ccb7c0 thru 0x8ccd22b (0x1a6c bytes) 2024-04-24T00:36:02.598 INFO:teuthology.orchestra.run.smithi064.stdout:3763 punch from 0xde1d96a to 0xde2d406, (0xfa9c bytes) 2024-04-24T00:36:02.598 INFO:teuthology.orchestra.run.smithi064.stdout:3764 write 0x84e41a6 thru 0x84e7eac (0x3d07 bytes) 2024-04-24T00:36:02.602 INFO:teuthology.orchestra.run.smithi064.stdout:3765 punch from 0xbdfcb80 to 0xbe026cd, (0x5b4d bytes) 2024-04-24T00:36:02.602 INFO:teuthology.orchestra.run.smithi064.stdout:3766 write 0xa0446a4 thru 0xa053f91 (0xf8ee bytes) 2024-04-24T00:36:02.612 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:02.607+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:02.613 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:02.613 INFO:teuthology.orchestra.run.smithi064.stdout:3768 read 0x5cc1855 thru 0x5cc2c23 (0x13cf bytes) 2024-04-24T00:36:02.614 INFO:teuthology.orchestra.run.smithi064.stdout:3769 write 0xb333b85 thru 0xb338f5b (0x53d7 bytes) 2024-04-24T00:36:02.618 INFO:teuthology.orchestra.run.smithi064.stdout:3771 writesame 0xcf72e97 thru 0xcf7e7c8 data_size 0x37(0xb932 bytes) 2024-04-24T00:36:02.649 INFO:teuthology.orchestra.run.smithi064.stdout:3772 trunc from 0xe125dc6 to 0x2ba9b62 2024-04-24T00:36:02.682 INFO:teuthology.orchestra.run.smithi064.stdout:3773 read 0xd979fd thru 0xda179e (0x9da2 bytes) 2024-04-24T00:36:02.695 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 0.96239 (f65f) 2024-04-24T00:36:02.713 INFO:teuthology.orchestra.run.smithi064.stdout:3775 writesame 0xcf100d6 thru 0xcf19bed data_size 0x1c(0x9b18 bytes) 2024-04-24T00:36:02.713 INFO:teuthology.orchestra.run.smithi064.stdout:3777 read 0x8ef31e3 thru 0x8ef5e82 (0x2ca0 bytes) 2024-04-24T00:36:02.713 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:02.707+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:3778 trunc from 0xcf19bee to 0x6c946b0 2024-04-24T00:36:02.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:02.742 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:02.742 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:02.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:02.846 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:02.840+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:02.885 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:02.886 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:02.901 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:02.914 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:02.911+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:02.917 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:02.918 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.703990635, 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.70363972, 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.6967514, 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.438469773999998, 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:02.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:02.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.428193358000001, 2024-04-24T00:36:02.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:02.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.966242555000001, 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.943 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:02.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.953834289, 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.949 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:02.953 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.953802535000001, 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81732 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2539520~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.950353678999999, 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042008566999999997, 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81732 2024-04-24T00:36:02.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965400+0000", 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4150000000000002e-06 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965407+0000", 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0920000000000003e-06 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965411+0000", 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7099999999999998e-06 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007262+0000", 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041850738999999998 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007328+0000", 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6375000000000001e-05 2024-04-24T00:36:02.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007406+0000", 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7844999999999998e-05 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81733 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2818048~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.950298142000001, 2024-04-24T00:36:02.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042136290999999999, 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81733 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:02.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965454+0000", 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965457+0000", 2024-04-24T00:36:02.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3799999999999998e-06 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965466+0000", 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5280000000000003e-06 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965472+0000", 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4670000000000001e-06 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007435+0000", 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041963037000000002 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007502+0000", 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6727e-05 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007590+0000", 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7614999999999994e-05 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.950270038999999, 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:02.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:02.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.940208534, 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.726265627, 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.716253683000001, 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.442874821, 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:02.991 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.426206371999999, 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.240308589, 2024-04-24T00:36:02.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:02.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:02.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:03.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.55655023199999998, 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:03.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:03.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:03.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.53433323399999999, 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:03.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:03.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:03.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:03.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.37311195400000002, 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:03.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:03.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:03.017 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:03.018 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:03.019 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:03.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:03.019 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:03.020 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:03.020 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:03.020 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:03.020 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:03.021 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:03.021 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:03.021 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:03.021 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:03.012+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:03.119 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:03.116+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:03.151 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:03.152 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:03.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:03.220 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:03.212+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:03.289 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:03.289 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:03.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:03.350 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:03.347+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:03.422 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:03.419+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:03.423 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:03.424 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:03.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:03.526 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:03.523+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:03.608 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:03.608 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:03.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:03.624 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:03.620+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:03.725 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:03.716+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:03.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:36:03.786 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:03.787 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:03.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:03.846 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:03.843+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:03.920 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:03.921 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:03.926 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:03.920+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:03.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:04.027 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:04.020+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:04.102 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:04.102 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:04.116 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:04.166 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:04.163+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:04.228 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:04.220+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:04.236 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:04.236 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:04.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:04.338 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:04.335+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:04.418 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:04.418 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:04.429 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:04.424+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:04.432 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:04.530 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:04.524+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:04.565 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:36:04.602 INFO:teuthology.orchestra.run.smithi064.stdout:3779 write 0xcf300 thru 0xd0414 (0x1115 bytes) 2024-04-24T00:36:04.606 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:04.606 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:04.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:04.627 INFO:teuthology.orchestra.run.smithi064.stdout:3780 write 0xbcfc838 thru 0xbd01b23 (0x52ec bytes) 2024-04-24T00:36:04.633 INFO:teuthology.orchestra.run.smithi064.stdout:3781 read 0x2a4018c thru 0x2a4f799 (0xf60e bytes) 2024-04-24T00:36:04.633 INFO:teuthology.orchestra.run.smithi064.stdout:3782 punch from 0x1433c23 to 0x143f24b, (0xb628 bytes) 2024-04-24T00:36:04.633 INFO:teuthology.orchestra.run.smithi064.stdout:3784 punch from 0xa86fd48 to 0xa8786c6, (0x897e bytes) 2024-04-24T00:36:04.634 INFO:teuthology.orchestra.run.smithi064.stdout:3785 read 0x788e415 thru 0x78924a8 (0x4094 bytes) 2024-04-24T00:36:04.634 INFO:teuthology.orchestra.run.smithi064.stdout:3786 punch from 0xae5b04 to 0xaeed7a, (0x9276 bytes) 2024-04-24T00:36:04.634 INFO:teuthology.orchestra.run.smithi064.stdout:3787 read 0x66d9572 thru 0x66e19d8 (0x8467 bytes) 2024-04-24T00:36:04.649 INFO:teuthology.orchestra.run.smithi064.stdout:3788 write 0xe30f6be thru 0xe311c04 (0x2547 bytes) 2024-04-24T00:36:04.654 INFO:teuthology.orchestra.run.smithi064.stdout:3789 punch from 0xb6e9862 to 0xb6f716b, (0xd909 bytes) 2024-04-24T00:36:04.654 INFO:teuthology.orchestra.run.smithi064.stdout:3790 read 0x2a6e160 thru 0x2a731e1 (0x5082 bytes) 2024-04-24T00:36:04.654 INFO:teuthology.orchestra.run.smithi064.stdout:3791 punch from 0x5e97999 to 0x5e9941a, (0x1a81 bytes) 2024-04-24T00:36:04.674 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:04.671+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:04.678 INFO:teuthology.orchestra.run.smithi064.stdout:3792 trunc from 0xe311c05 to 0x1dad5cf 2024-04-24T00:36:04.713 INFO:teuthology.orchestra.run.smithi064.stdout:3793 write 0x2c74c4a thru 0x2c81e2f (0xd1e6 bytes) 2024-04-24T00:36:04.715 INFO:teuthology.orchestra.run.smithi064.stdout:3796 write 0xa63e1 thru 0xb5de1 (0xfa01 bytes) 2024-04-24T00:36:04.722 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:04.722 INFO:teuthology.orchestra.run.smithi064.stdout:3798 read 0x4ac5ed thru 0x4b8c9f (0xc6b3 bytes) 2024-04-24T00:36:04.732 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:04.727+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:04.743 INFO:teuthology.orchestra.run.smithi064.stdout:3799 write 0x7a2851e thru 0x7a2b6a9 (0x318c bytes) 2024-04-24T00:36:04.745 INFO:teuthology.orchestra.run.smithi064.stdout:3800 punch from 0x41b349d to 0x41b5cef, (0x2852 bytes) 2024-04-24T00:36:04.746 INFO:teuthology.orchestra.run.smithi064.stdout:3802 punch from 0x3728390 to 0x372caf1, (0x4761 bytes) 2024-04-24T00:36:04.746 INFO:teuthology.orchestra.run.smithi064.stdout:3803 write 0x30aef80 thru 0x30b27d1 (0x3852 bytes) 2024-04-24T00:36:04.748 INFO:teuthology.orchestra.run.smithi064.stdout:3804 read 0x47396ea thru 0x4740e1e (0x7735 bytes) 2024-04-24T00:36:04.749 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:04.749 INFO:teuthology.orchestra.run.smithi064.stdout:3806 punch from 0x1d23519 to 0x1d271a1, (0x3c88 bytes) 2024-04-24T00:36:04.749 INFO:teuthology.orchestra.run.smithi064.stdout:3807 punch from 0x4e96449 to 0x4ea2e4d, (0xca04 bytes) 2024-04-24T00:36:04.749 INFO:teuthology.orchestra.run.smithi064.stdout:3808 write 0x18817d2 thru 0x188c346 (0xab75 bytes) 2024-04-24T00:36:04.761 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:04.761 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:04.773 INFO:teuthology.orchestra.run.smithi064.stdout:3811 write 0xaf862bb thru 0xaf8c4d6 (0x621c bytes) 2024-04-24T00:36:04.775 INFO:teuthology.orchestra.run.smithi064.stdout:3812 read 0xa8a0d7b thru 0xa8aaa59 (0x9cdf bytes) 2024-04-24T00:36:04.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:04.782 INFO:teuthology.orchestra.run.smithi064.stdout:3813 trunc from 0xaf8c4d7 to 0x8d2b6d0 2024-04-24T00:36:04.801 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:04.801 INFO:teuthology.orchestra.run.smithi064.stdout:3815 write 0x6bafdd2 thru 0x6bbdf2a (0xe159 bytes) 2024-04-24T00:36:04.810 INFO:teuthology.orchestra.run.smithi064.stdout:3816 read 0x5c77947 thru 0x5c7ce62 (0x551c bytes) 2024-04-24T00:36:04.811 INFO:teuthology.orchestra.run.smithi064.stdout:3818 read 0x4d7fee0 thru 0x4d86cc1 (0x6de2 bytes) 2024-04-24T00:36:04.812 INFO:teuthology.orchestra.run.smithi064.stdout:3819 read 0x14544b7 thru 0x145bd4d (0x7897 bytes) 2024-04-24T00:36:04.813 INFO:teuthology.orchestra.run.smithi064.stdout:3822 read 0x5898bfe thru 0x58a7c5f (0xf062 bytes) 2024-04-24T00:36:04.814 INFO:teuthology.orchestra.run.smithi064.stdout:3823 read 0x7a70504 thru 0x7a75196 (0x4c93 bytes) 2024-04-24T00:36:04.815 INFO:teuthology.orchestra.run.smithi064.stdout:3824 write 0x14d2c9c thru 0x14dbef2 (0x9257 bytes) 2024-04-24T00:36:04.822 INFO:teuthology.orchestra.run.smithi064.stdout:3827 read 0x123a83c thru 0x124667d (0xbe42 bytes) 2024-04-24T00:36:04.822 INFO:teuthology.orchestra.run.smithi064.stdout:3828 read 0x36c242 thru 0x3770e9 (0xaea8 bytes) 2024-04-24T00:36:04.822 INFO:teuthology.orchestra.run.smithi064.stdout:3829 read 0x162b640 thru 0x16349b7 (0x9378 bytes) 2024-04-24T00:36:04.823 INFO:teuthology.orchestra.run.smithi064.stdout:3830 punch from 0x30f8405 to 0x30ff6ca, (0x72c5 bytes) 2024-04-24T00:36:04.823 INFO:teuthology.orchestra.run.smithi064.stdout:3831 writesame 0x2bd48d1 thru 0x2bda1be data_size 0x1(0x58ee bytes) 2024-04-24T00:36:04.833 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:04.828+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:04.911 INFO:teuthology.orchestra.run.smithi064.stdout:3832 trunc from 0x8d2b6d0 to 0x1e2b3f2 2024-04-24T00:36:04.928 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:04.934 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:04.928+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:04.945 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:04.945 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:04.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:04.961 INFO:teuthology.orchestra.run.smithi064.stdout:3836 write 0x939d5f6 thru 0x93abae7 (0xe4f2 bytes) 2024-04-24T00:36:04.967 INFO:teuthology.orchestra.run.smithi064.stdout:3837 trunc from 0x93abae8 to 0x874ed75 2024-04-24T00:36:04.986 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:04.987 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:04.987 INFO:teuthology.orchestra.run.smithi064.stdout:3842 writesame 0x41bf07a thru 0x41c5d7f data_size 0x5(0x6d06 bytes) 2024-04-24T00:36:05.009 INFO:teuthology.orchestra.run.smithi064.stdout:3844 trunc from 0x874ed75 to 0xc2d1928 2024-04-24T00:36:05.020 INFO:teuthology.orchestra.run.smithi064.stdout:3845 write 0x67f40b3 thru 0x6800cab (0xcbf9 bytes) 2024-04-24T00:36:05.034 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:05.028+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:05.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:05.042 INFO:teuthology.orchestra.run.smithi064.stdout:3846 write 0xd8f3695 thru 0xd8f5c88 (0x25f4 bytes) 2024-04-24T00:36:05.048 INFO:teuthology.orchestra.run.smithi064.stdout:3848 read 0x8e12c24 thru 0x8e13ac1 (0xe9e bytes) 2024-04-24T00:36:05.063 INFO:teuthology.orchestra.run.smithi064.stdout:3849 writesame 0xdf3bcfe thru 0xdf40d59 data_size 0x4(0x505c bytes) 2024-04-24T00:36:05.063 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:05.063 INFO:teuthology.orchestra.run.smithi064.stdout:3852 write 0x7155dac thru 0x715617c (0x3d1 bytes) 2024-04-24T00:36:05.065 INFO:teuthology.orchestra.run.smithi064.stdout:3853 read 0x4a8a086 thru 0x4a96e18 (0xcd93 bytes) 2024-04-24T00:36:05.066 INFO:teuthology.orchestra.run.smithi064.stdout:3854 read 0xc9682d0 thru 0xc9721bd (0x9eee bytes) 2024-04-24T00:36:05.066 INFO:teuthology.orchestra.run.smithi064.stdout:3857 read 0x85920d4 thru 0x8596b41 (0x4a6e bytes) 2024-04-24T00:36:05.067 INFO:teuthology.orchestra.run.smithi064.stdout:3858 punch from 0x14f3d06 to 0x14f6d22, (0x301c bytes) 2024-04-24T00:36:05.067 INFO:teuthology.orchestra.run.smithi064.stdout:3859 write 0x22d9e8 thru 0x23aae5 (0xd0fe bytes) 2024-04-24T00:36:05.080 INFO:teuthology.orchestra.run.smithi064.stdout:3861 read 0x11ef11c thru 0x11f7eea (0x8dcf bytes) 2024-04-24T00:36:05.080 INFO:teuthology.orchestra.run.smithi064.stdout:3862 punch from 0xc81ccf to 0xc8dd67, (0xc098 bytes) 2024-04-24T00:36:05.101 INFO:teuthology.orchestra.run.smithi064.stdout:3863 trunc from 0xdf40d5a to 0x2967e5 2024-04-24T00:36:05.110 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:05.111 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:05.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:05.136 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:05.131+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:05.174 INFO:teuthology.orchestra.run.smithi064.stdout:3864 punch from 0x136b38 to 0x144b64, (0xe02c bytes) 2024-04-24T00:36:05.190 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:05.190 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:05.190 INFO:teuthology.orchestra.run.smithi064.stdout:3865 trunc from 0x2967e5 to 0xb0ab0c3 2024-04-24T00:36:05.204 INFO:teuthology.orchestra.run.smithi064.stdout:3866 trunc from 0xb0ab0c3 to 0x9eb094c 2024-04-24T00:36:05.208 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:05.208 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:05.219 INFO:teuthology.orchestra.run.smithi064.stdout:3867 read 0x27e65a1 thru 0x27efffa (0x9a5a bytes) 2024-04-24T00:36:05.226 INFO:teuthology.orchestra.run.smithi064.stdout:3870 trunc from 0x9eb094c to 0xe77f5aa 2024-04-24T00:36:05.226 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:05.226 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:05.236 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:05.228+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:05.244 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:05.244 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:05.249 INFO:teuthology.orchestra.run.smithi064.stdout:3872 trunc from 0xe77f5aa to 0x756a4c9 2024-04-24T00:36:05.265 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:05.266 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:05.286 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:05.286 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:05.288 INFO:teuthology.orchestra.run.smithi064.stdout:3874 writesame 0xb20c8b5 thru 0xb217564 data_size 0x48(0xacb0 bytes) 2024-04-24T00:36:05.288 INFO:teuthology.orchestra.run.smithi064.stdout:3876 read 0x7632a63 thru 0x763a5f5 (0x7b93 bytes) 2024-04-24T00:36:05.288 INFO:teuthology.orchestra.run.smithi064.stdout:3877 writesame 0xa49974f thru 0xa49bd3f data_size 0x1(0x25f1 bytes) 2024-04-24T00:36:05.299 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:05.300 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:05.310 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:05.310 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:05.311 INFO:teuthology.orchestra.run.smithi064.stdout:3878 write 0xb2a81ce thru 0xb2b125c (0x908f bytes) 2024-04-24T00:36:05.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:05.319 INFO:teuthology.orchestra.run.smithi064.stdout:3879 punch from 0x211eca3 to 0x2125950, (0x6cad bytes) 2024-04-24T00:36:05.319 INFO:teuthology.orchestra.run.smithi064.stdout:3880 punch from 0x1f8eac6 to 0x1f91534, (0x2a6e bytes) 2024-04-24T00:36:05.319 INFO:teuthology.orchestra.run.smithi064.stdout:3881 write 0x3dd564d thru 0x3dd6efa (0x18ae bytes) 2024-04-24T00:36:05.321 INFO:teuthology.orchestra.run.smithi064.stdout:3882 read 0xd4f0ee thru 0xd54ef9 (0x5e0c bytes) 2024-04-24T00:36:05.322 INFO:teuthology.orchestra.run.smithi064.stdout:3883 read 0x38aa11a thru 0x38ac2c0 (0x21a7 bytes) 2024-04-24T00:36:05.323 INFO:teuthology.orchestra.run.smithi064.stdout:3884 read 0x9a883e7 thru 0x9a8f4e9 (0x7103 bytes) 2024-04-24T00:36:05.333 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:05.333 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:05.337 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:05.332+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:05.338 INFO:teuthology.orchestra.run.smithi064.stdout:3885 write 0xbd831ea thru 0xbd85f46 (0x2d5d bytes) 2024-04-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi064.stdout:3886 trunc from 0xbd85f47 to 0x3baf7b8 2024-04-24T00:36:05.403 INFO:teuthology.orchestra.run.smithi064.stdout:3888 write 0xb20c23b thru 0xb212dcd (0x6b93 bytes) 2024-04-24T00:36:05.406 INFO:teuthology.orchestra.run.smithi064.stdout:3889 read 0x2da3f0 thru 0x2e9b1c (0xf72d bytes) 2024-04-24T00:36:05.408 INFO:teuthology.orchestra.run.smithi064.stdout:3890 writesame 0x85059a6 thru 0x8507bbe data_size 0x7(0x2219 bytes) 2024-04-24T00:36:05.428 INFO:teuthology.orchestra.run.smithi064.stdout:3891 write 0xc898e5a thru 0xc89f44d (0x65f4 bytes) 2024-04-24T00:36:05.435 INFO:teuthology.orchestra.run.smithi064.stdout:3892 read 0x39d74d thru 0x3a4ef7 (0x77ab bytes) 2024-04-24T00:36:05.435 INFO:teuthology.orchestra.run.smithi064.stdout:3893 punch from 0x8e47400 to 0x8e52e4c, (0xba4c bytes) 2024-04-24T00:36:05.435 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:05.435 INFO:teuthology.orchestra.run.smithi064.stdout:3896 read 0xc72605b thru 0xc72e3be (0x8364 bytes) 2024-04-24T00:36:05.439 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:05.432+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:05.442 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-04-24T00:36:05.442 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:05.451 INFO:teuthology.orchestra.run.smithi064.stdout:3897 write 0xe2b4471 thru 0xe2b8566 (0x40f6 bytes) 2024-04-24T00:36:05.459 INFO:teuthology.orchestra.run.smithi064.stdout:3899 write 0xd7ff07 thru 0xd869f9 (0x6af3 bytes) 2024-04-24T00:36:05.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:05.470 INFO:teuthology.orchestra.run.smithi064.stdout:3900 trunc from 0xe2b8567 to 0xb30f483 2024-04-24T00:36:05.499 INFO:teuthology.orchestra.run.smithi064.stdout:3901 punch from 0xb2db60f to 0xb2e4c7b, (0x966c bytes) 2024-04-24T00:36:05.499 INFO:teuthology.orchestra.run.smithi064.stdout:3902 write 0x1ddbd21 thru 0x1dea4ae (0xe78e bytes) 2024-04-24T00:36:05.509 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:05.510 INFO:teuthology.orchestra.run.smithi064.stdout:3905 punch from 0xb06bfd3 to 0xb079110, (0xd13d bytes) 2024-04-24T00:36:05.510 INFO:teuthology.orchestra.run.smithi064.stdout:3906 write 0x119d947 thru 0x11acad8 (0xf192 bytes) 2024-04-24T00:36:05.515 INFO:teuthology.orchestra.run.smithi064.stdout:3907 read 0x8a2c28c thru 0x8a337fa (0x756f bytes) 2024-04-24T00:36:05.515 INFO:teuthology.orchestra.run.smithi064.stdout:3908 writesame 0x4abdb5 thru 0x4b7497 data_size 0x1(0xb6e3 bytes) 2024-04-24T00:36:05.540 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:05.532+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:05.599 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:05.599 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:05.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:05.658 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:05.652+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:05.672 INFO:teuthology.orchestra.run.smithi064.stdout:3911 writesame 0x9c05a45 thru 0x9c0ed51 data_size 0x1(0x930d bytes) 2024-04-24T00:36:05.742 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:05.739+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:05.784 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:05.784 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:05.784 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:05.784 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:05.784 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:05.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:05.815 INFO:teuthology.orchestra.run.smithi064.stdout:3912 write 0xb8f9cb7 thru 0xb8fa87b (0xbc5 bytes) 2024-04-24T00:36:05.822 INFO:teuthology.orchestra.run.smithi064.stdout:3913 trunc from 0xb8fa87c to 0xe432601 2024-04-24T00:36:05.834 INFO:teuthology.orchestra.run.smithi064.stdout:3915 read 0x3339b87 thru 0x333dad8 (0x3f52 bytes) 2024-04-24T00:36:05.834 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:05.845 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:05.836+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:05.848 INFO:teuthology.orchestra.run.smithi064.stdout:3917 trunc from 0xe432601 to 0x693f814 2024-04-24T00:36:05.944 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:05.936+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:05.951 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:05.951 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:05.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.512375621000004, 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.952 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.456864135000004, 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:36:05.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:36:05.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.356287487000003, 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:05.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:36:05.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:05.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.314722532999994, 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:36:05.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:36:05.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.750990971, 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:36:05.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:36:05.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.248017574000002, 2024-04-24T00:36:05.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:05.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:36:05.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.009703340999998, 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:05.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:05.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:36:05.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974762918, 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:36:05.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:36:05.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:05.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:05.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974742044999999, 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:36:05.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:05.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974718666000001, 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:05.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:36:05.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:36:06.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974697123999999, 2024-04-24T00:36:06.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:06.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974675604000002, 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.006 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:36:06.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974629447000002, 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974605327999999, 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.014 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:06.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:36:06.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974581923999999, 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974554151, 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:36:06.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974532738000001, 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:36:06.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:36:06.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.974512146999999, 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:36:06.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.758441905000002, 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:36:06.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:36:06.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:36:06.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:06.042 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.730471001000002, 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:06.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:36:06.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:36:06.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.046 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:06.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:06.051 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:06.044+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:06.152 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:06.144+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:06.222 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:06.222 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:06.222 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:06.223 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:06.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:06.278 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:06.272+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:06.356 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:06.351+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:06.406 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:06.406 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:06.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:06.465 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:06.460+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:06.550 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21434 3.5 3:a702b6a2:::rbd_data.115eeb9272a5.0000000000000007:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.962541775000005, 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070638200000000002, 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.551 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21434 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583081+0000", 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583083+0000", 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5799999999999999e-06 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583092+0000", 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1640000000000007e-06 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583117+0000", 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5287999999999999e-05 2024-04-24T00:36:06.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583127+0000", 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5489999999999995e-06 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584028+0000", 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090124499999999995 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584117+0000", 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9065000000000002e-05 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584236+0000", 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119011 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.587283+0000", 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030466730000000002 2024-04-24T00:36:06.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590084+0000", 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028009300000000001 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590117+0000", 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3584999999999998e-05 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590145+0000", 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.773e-05 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.962467919999995, 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.265773136000007, 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.460241177, 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:36:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:77813 3.0 e102/21 3:05afdb1e:::rbd_data.115e805f80b3.0000000000000004:head v 102'237, mlcod=102'235)", 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.880800827999998, 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074761180000000003, 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.571 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664825+0000", 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-04-24T00:36:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664911+0000", 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5873999999999998e-05 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664944+0000", 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664959+0000", 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5979000000000001e-05 2024-04-24T00:36:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665112+0000", 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152943 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665157+0000", 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4725000000000003e-05 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672178+0000", 2024-04-24T00:36:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070206390000000004 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672298+0000", 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120308 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head, version: 45'133, data_included: [0~131072,196158~37038,262144~131072,524288~131072,786432~262144,1258308~52412,1441792~131072,3014656~131072,3267061~6628,6160384~34994], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head@45'133, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 6195378, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.623561745000003, 2024-04-24T00:36:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011957290000000001, 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922063+0000", 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.306e-06 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923213+0000", 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011495310000000001 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923258+0000", 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4747999999999999e-05 2024-04-24T00:36:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923284+0000", 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5873000000000001e-05 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923436+0000", 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015226500000000001 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923452+0000", 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6022e-05 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.934018+0000", 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010566545 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head, version: 45'137, data_included: [0~131072,262144~131072,524288~131072,786432~199751,1179648~1552,1441792~3851,3014656~131072,5775043~64169,6160384~131072], data_size: 924683, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head@45'137, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8781824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.594737023, 2024-04-24T00:36:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011581296, 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950888+0000", 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951161+0000", 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027280499999999999 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951196+0000", 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123999999999998e-05 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951224+0000", 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8045e-05 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951356+0000", 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013187799999999999 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951376+0000", 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.003e-05 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.962467+0000", 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011091474 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.389402208, 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:06.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head, version: 80'319, data_included: [0~393216,524288~131072,1179648~524288], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head@80'319, size: 13631488, copy_subset: [(0, 13631488)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.376648115000002, 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089307889999999997, 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168976+0000", 2024-04-24T00:36:06.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169620+0000", 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064358299999999998 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169650+0000", 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0111e-05 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169671+0000", 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1231000000000001e-05 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169789+0000", 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118368 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169803+0000", 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3779e-05 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.177905+0000", 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081024800000000004 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.364501169, 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:06.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.347757010999999, 2024-04-24T00:36:06.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:06.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.257608017999999, 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:06.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89926 3.12 3:49ee3472:::rbd_object_map.115e56aa02fc.0000000000000031:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.862150694, 2024-04-24T00:36:06.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068252479999999999, 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89926 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683472+0000", 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683473+0000", 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8899999999999998e-07 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683491+0000", 2024-04-24T00:36:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7829000000000002e-05 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683506+0000", 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5457000000000001e-05 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.683520+0000", 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3186e-05 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.684785+0000", 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001265544 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.684895+0000", 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010945 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.685272+0000", 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037777200000000001 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.687975+0000", 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027025759999999999 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690177+0000", 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002202196 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690241+0000", 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4312000000000004e-05 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.690297+0000", 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5937000000000003e-05 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89927 3.12 3:49ee3472:::rbd_object_map.115e56aa02fc.0000000000000031:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.859058925999999, 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068456009999999998, 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89927 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686564+0000", 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686566+0000", 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2079999999999999e-06 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686575+0000", 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8359999999999998e-06 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686622+0000", 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7533e-05 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686633+0000", 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0546e-05 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686726+0000", 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3431999999999996e-05 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.686834+0000", 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010771 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.687193+0000", 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035882399999999999 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.689501+0000", 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023082760000000002 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693281+0000", 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037798509999999999 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693348+0000", 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6913000000000001e-05 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.693409+0000", 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1471999999999997e-05 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.616 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.850451338999999, 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:06.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:06.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:06.622 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.840064642000002, 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.626 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.831121353, 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:06.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:06.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:90104 3.18 3:193b8512:::rbd_data.115e56aa02fc.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write 1644318~44706] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.43498846, 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084175689999999997, 2024-04-24T00:36:06.631 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 90104 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110636+0000", 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110706+0000", 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9392000000000002e-05 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110734+0000", 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8538999999999999e-05 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110747+0000", 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860000000000001e-05 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110850+0000", 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102988 2024-04-24T00:36:06.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111107+0000", 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025716600000000002 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111346+0000", 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000238373 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.117684+0000", 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063379949999999999 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118724+0000", 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001040685 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118914+0000", 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018984699999999999 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.119052+0000", 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137701 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPushReply(3.12 128/127 [PushReplyOp(3:4e045fb2:::rbd_data.115e6f07caf6.0000000000000007:head)])", 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.167799090999999, 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089128709999999993, 2024-04-24T00:36:06.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377827+0000", 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5870000000000001e-06 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377852+0000", 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5083e-05 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377890+0000", 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7279e-05 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377914+0000", 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4496000000000001e-05 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.378154+0000", 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024028 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.386737+0000", 2024-04-24T00:36:06.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085821460000000006 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91841 3.0 3:02b978df:::rbd_data.115ee57427d8.0000000000000008:head [write 6244202~37028] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.590049+0000", 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9555735489999999, 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070732210000000002, 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:06.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91841 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590049+0000", 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590049+0000", 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590052+0000", 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5940000000000001e-06 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590085+0000", 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2904999999999998e-05 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590108+0000", 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3649999999999999e-05 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:06.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590120+0000", 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1909999999999999e-05 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590285+0000", 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016510599999999999 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590572+0000", 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028664700000000001 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.591013+0000", 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044140999999999999 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.596705+0000", 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056917160000000003 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.597034+0000", 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000329191 2024-04-24T00:36:06.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.597080+0000", 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5484e-05 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.597122+0000", 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2608000000000003e-05 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:06.647 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:06.647 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:06.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:36:06.648 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:06.647+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:06.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:06.743 INFO:teuthology.orchestra.run.smithi064.stdout:3918 punch from 0x4857bf2 to 0x4861b33, (0x9f41 bytes) 2024-04-24T00:36:06.743 INFO:teuthology.orchestra.run.smithi064.stdout:3921 read 0x65e7e63 thru 0x65f1757 (0x98f5 bytes) 2024-04-24T00:36:06.744 INFO:teuthology.orchestra.run.smithi064.stdout:3922 read 0x51c4009 thru 0x51c7c5b (0x3c53 bytes) 2024-04-24T00:36:06.744 INFO:teuthology.orchestra.run.smithi064.stdout:3923 punch from 0x4dca4ad to 0x4dd2fbb, (0x8b0e bytes) 2024-04-24T00:36:06.744 INFO:teuthology.orchestra.run.smithi064.stdout:3924 punch from 0x598ebb1 to 0x5990600, (0x1a4f bytes) 2024-04-24T00:36:06.749 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:06.740+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:06.753 INFO:teuthology.orchestra.run.smithi064.stdout:3925 trunc from 0x693f814 to 0xe71f1c 2024-04-24T00:36:06.794 INFO:teuthology.orchestra.run.smithi064.stdout:3927 write 0x1da387b thru 0x1dadcd3 (0xa459 bytes) 2024-04-24T00:36:06.797 INFO:teuthology.orchestra.run.smithi064.stdout:3929 read 0x3eaa7b thru 0x3f07b4 (0x5d3a bytes) 2024-04-24T00:36:06.826 INFO:teuthology.orchestra.run.smithi064.stdout:3930 write 0xc87afdd thru 0xc8891f3 (0xe217 bytes) 2024-04-24T00:36:06.828 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:06.828 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:06.828 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:06.829 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:06.834 INFO:teuthology.orchestra.run.smithi064.stdout:3931 write 0x20cd05 thru 0x215b6a (0x8e66 bytes) 2024-04-24T00:36:06.841 INFO:teuthology.orchestra.run.smithi064.stdout:3932 read 0x5ebc184 thru 0x5ec9ff1 (0xde6e bytes) 2024-04-24T00:36:06.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:06.850 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:06.847+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:06.857 INFO:teuthology.orchestra.run.smithi064.stdout:3933 write 0xd8a83dd thru 0xd8b7663 (0xf287 bytes) 2024-04-24T00:36:06.863 INFO:teuthology.orchestra.run.smithi064.stdout:3934 writesame 0x20a3344 thru 0x20b1d47 data_size 0x22(0xea04 bytes) 2024-04-24T00:36:06.870 INFO:teuthology.orchestra.run.smithi064.stdout:3935 punch from 0x3318ffa to 0x33201dd, (0x71e3 bytes) 2024-04-24T00:36:06.870 INFO:teuthology.orchestra.run.smithi064.stdout:3936 read 0x11d0063 thru 0x11d3c3f (0x3bdd bytes) 2024-04-24T00:36:06.870 INFO:teuthology.orchestra.run.smithi064.stdout:3938 read 0x5c170ad thru 0x5c182bc (0x1210 bytes) 2024-04-24T00:36:06.870 INFO:teuthology.orchestra.run.smithi064.stdout:3939 write 0xd4407d2 thru 0xd44b4fe (0xad2d bytes) 2024-04-24T00:36:06.877 INFO:teuthology.orchestra.run.smithi064.stdout:3940 write 0xba3c661 thru 0xba3e1fb (0x1b9b bytes) 2024-04-24T00:36:06.884 INFO:teuthology.orchestra.run.smithi064.stdout:3941 read 0x451613a thru 0x45237bd (0xd684 bytes) 2024-04-24T00:36:06.885 INFO:teuthology.orchestra.run.smithi064.stdout:3942 write 0x230ecf4 thru 0x2315d01 (0x700e bytes) 2024-04-24T00:36:06.895 INFO:teuthology.orchestra.run.smithi064.stdout:3943 write 0xa543305 thru 0xa54389a (0x596 bytes) 2024-04-24T00:36:06.898 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:06.898 INFO:teuthology.orchestra.run.smithi064.stdout:3946 write 0x65a524f thru 0x65a5d5f (0xb11 bytes) 2024-04-24T00:36:06.900 INFO:teuthology.orchestra.run.smithi064.stdout:3948 writesame 0x364e5fd thru 0x3651f2c data_size 0x50(0x3930 bytes) 2024-04-24T00:36:06.903 INFO:teuthology.orchestra.run.smithi064.stdout:3949 writesame 0xa3cb1f4 thru 0xa3d39b6 data_size 0x23(0x87c3 bytes) 2024-04-24T00:36:06.919 INFO:teuthology.orchestra.run.smithi064.stdout:3950 trunc from 0xd8b7664 to 0x96588ef 2024-04-24T00:36:06.951 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:06.947+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:06.951 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:06.951 INFO:teuthology.orchestra.run.smithi064.stdout:3952 read 0x32ad1c5 thru 0x32aebea (0x1a26 bytes) 2024-04-24T00:36:06.952 INFO:teuthology.orchestra.run.smithi064.stdout:3953 write 0x63b5f8e thru 0x63bfb1b (0x9b8e bytes) 2024-04-24T00:36:06.963 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:06.963 INFO:teuthology.orchestra.run.smithi064.stdout:3957 writesame 0x58962ed thru 0x589f06a data_size 0x3(0x8d7e bytes) 2024-04-24T00:36:06.978 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:06.978 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:06.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:07.018 INFO:teuthology.orchestra.run.smithi064.stdout:3959 read 0x7451ce thru 0x751804 (0xc637 bytes) 2024-04-24T00:36:07.020 INFO:teuthology.orchestra.run.smithi064.stdout:3960 clone 26 order 22 su 131072 sc 12 2024-04-24T00:36:07.053 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:07.047+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.91910496600001, 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:07.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:07.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.91875405099999, 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:07.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.91186573100001, 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:07.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:07.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 62.653584105, 2024-04-24T00:36:07.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:07.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:07.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 62.643307688999997, 2024-04-24T00:36:07.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:07.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.181356886000003, 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:07.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:07.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.168948620000002, 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:07.168 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.168916866000004, 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:07.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:07.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81732 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2539520~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.165468009999998, 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042008566999999997, 2024-04-24T00:36:07.177 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81732 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965400+0000", 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4150000000000002e-06 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965407+0000", 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0920000000000003e-06 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965411+0000", 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7099999999999998e-06 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007262+0000", 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041850738999999998 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007328+0000", 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6375000000000001e-05 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007406+0000", 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7844999999999998e-05 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.181 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81733 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2818048~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.165412473000004, 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042136290999999999, 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81733 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965454+0000", 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965457+0000", 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3799999999999998e-06 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965466+0000", 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5280000000000003e-06 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965472+0000", 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4670000000000001e-06 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007435+0000", 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041963037000000002 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007502+0000", 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6727e-05 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007590+0000", 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7614999999999994e-05 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:07.186 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.165384369999998, 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.155322865000002, 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:07.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:07.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.194 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.941379957999999, 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.931368014, 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:07.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.657989151999999, 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:07.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:07.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:07.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.641320703000002, 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.45542292, 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:07.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:07.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:07.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7716645629999999, 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7494475649999996, 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:07.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5882262850000002, 2024-04-24T00:36:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:07.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:07.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:07.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:07.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:07.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:07.234 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:07.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:07.236 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:07.228+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:07.337 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:07.331+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:07.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:07.367 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:07.367 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:07.367 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:07.367 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:07.381 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:07.438 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:07.432+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:07.508 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:07.509 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:07.524 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:07.525 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:07.539 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:07.535+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:07.540 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:07.540 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:07.552 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:07.552 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:07.552 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:07.556 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:07.556 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:07.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:07.576 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:07.576 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:07.594 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:07.594 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:07.611 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:07.631 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:07.631 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:07.640 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:07.632+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:07.727 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:07.727 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:07.727 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T00:36:07.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T00:36:07.736 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:36:07.737 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:07.737 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:07.741 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:07.732+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:07.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:07.842 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:07.839+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:07.880 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:07.880 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:07.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:07.943 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:07.936+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 144, 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-04-24T00:33:24.857711+0000", 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-04-24T00:36:07.719305+0000", 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-04-24T00:35:14.574643+0000", 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-04-24T00:35:24.656453+0000", 2024-04-24T00:36:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5806080, 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 5, 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T00:36:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 3, 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:39.734814+0000", 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:36:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-04-24T00:36:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "17", 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 8, 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 8, 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.25, 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:43.786311+0000", 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "20", 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.5, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.5, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "pool_client.0", 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:51.288702+0000", 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "144", 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 54, 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 144, 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.75, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.75, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.059999942779541, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.059999942779541, 2024-04-24T00:36:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1c80b3f4-99ee-4725-90a8-920116161bf5", 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 13, 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 74, 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 95, 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 53, 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 75, 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6817", 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6818", 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6819", 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6822", 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6823", 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6820", 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6821", 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6817/218259947", 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6819/218259947", 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6823/218259947", 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6821/218259947", 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "20e1331a-f8a2-48fd-b2b8-b7a92f21d78e", 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.264190673828125, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 122, 2024-04-24T00:36:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6824", 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6825", 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6826", 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6827", 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6830", 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6831", 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6828", 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6829", 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6825/637720411", 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6827/637720411", 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6831/637720411", 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6829/637720411", 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a51557a0-9634-4a20-82b0-b54cdd36b166", 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 129, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-04-24T00:36:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-04-24T00:36:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-04-24T00:36:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/820024667", 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6811/820024667", 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6815/820024667", 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6813/820024667", 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "aafffa79-f532-4441-b1d6-f6441c5eebf8", 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.79296875, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.3392791748046875, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 123, 2024-04-24T00:36:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/1974534323", 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6803/1974534323", 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1974534323", 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1974534323", 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "78f05556-598e-4be2-b682-e3f58863eb82", 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.9623870849609375, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 130, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6816", 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6818", 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6820", 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6821", 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6826", 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6828", 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6823", 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6824", 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6818/1642543604", 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6821/1642543604", 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6828/1642543604", 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6824/1642543604", 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:36:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5", 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 142, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6808", 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6809", 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6810", 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6811", 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6814", 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6815", 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6812", 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6813", 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6809/172726194", 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6811/172726194", 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6815/172726194", 2024-04-24T00:36:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6813/172726194", 2024-04-24T00:36:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "6dad12bf-94de-482a-a402-49e67d5e5dc7", 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 123, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6800", 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6801", 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6802", 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6803", 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6806", 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6807", 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6804", 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6805", 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6801/2195746987", 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6803/2195746987", 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2195746987", 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6805/2195746987", 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e6734cfe-3c24-4ddd-8ef0-5ed450efe772", 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:36:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 62, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6817", 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6819", 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6822", 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6825", 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6830", 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6831", 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6827", 2024-04-24T00:36:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6829", 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6819/954791674", 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6825/954791674", 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6831/954791674", 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6829/954791674", 2024-04-24T00:36:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:36:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.509233+0000", 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 75 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.465696+0000", 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.405177+0000", 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.269202+0000", 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.359180+0000", 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.351834+0000", 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.490127+0000", 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.484842+0000", 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.4", 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.16", 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: 2 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.1b", 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-04-24T00:36:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [ 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 53, 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:08.115 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:08.117 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:08.117 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:08.118 INFO:tasks.thrashosds.thrasher:j is {'epoch': 144, 'fsid': '6a70d87d-4453-47be-af2a-5590f9851254', 'created': '2024-04-24T00:33:24.857711+0000', 'modified': '2024-04-24T00:36:07.719305+0000', 'last_up_change': '2024-04-24T00:35:14.574643+0000', 'last_in_change': '2024-04-24T00:35:24.656453+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T00:33:39.734814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T00:33:43.786311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-24T00:33:51.288702+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 54, 'snap_epoch': 144, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 0.75, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '1c80b3f4-99ee-4725-90a8-920116161bf5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 74, 'up_from': 95, 'up_thru': 53, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 218259947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 218259947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 218259947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 218259947}]}, 'public_addr': '172.21.15.64:6817/218259947', 'cluster_addr': '172.21.15.64:6819/218259947', 'heartbeat_back_addr': '172.21.15.64:6823/218259947', 'heartbeat_front_addr': '172.21.15.64:6821/218259947', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'up': 1, 'in': 1, 'weight': 0.264190673828125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 637720411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 637720411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 637720411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 637720411}]}, 'public_addr': '172.21.15.64:6825/637720411', 'cluster_addr': '172.21.15.64:6827/637720411', 'heartbeat_back_addr': '172.21.15.64:6831/637720411', 'heartbeat_front_addr': '172.21.15.64:6829/637720411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a51557a0-9634-4a20-82b0-b54cdd36b166', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 820024667}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 820024667}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 820024667}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 820024667}]}, 'public_addr': '172.21.15.64:6809/820024667', 'cluster_addr': '172.21.15.64:6811/820024667', 'heartbeat_back_addr': '172.21.15.64:6815/820024667', 'heartbeat_front_addr': '172.21.15.64:6813/820024667', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aafffa79-f532-4441-b1d6-f6441c5eebf8', 'up': 1, 'in': 1, 'weight': 0.79296875, 'primary_affinity': 0.3392791748046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1974534323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1974534323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1974534323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1974534323}]}, 'public_addr': '172.21.15.64:6801/1974534323', 'cluster_addr': '172.21.15.64:6803/1974534323', 'heartbeat_back_addr': '172.21.15.64:6807/1974534323', 'heartbeat_front_addr': '172.21.15.64:6805/1974534323', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '78f05556-598e-4be2-b682-e3f58863eb82', 'up': 1, 'in': 1, 'weight': 0.9623870849609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6816', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6818', 'nonce': 1642543604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6820', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6821', 'nonce': 1642543604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6826', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6828', 'nonce': 1642543604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6823', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6824', 'nonce': 1642543604}]}, 'public_addr': '172.21.15.167:6818/1642543604', 'cluster_addr': '172.21.15.167:6821/1642543604', 'heartbeat_back_addr': '172.21.15.167:6828/1642543604', 'heartbeat_front_addr': '172.21.15.167:6824/1642543604', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 142, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6808', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 172726194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6810', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 172726194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6814', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 172726194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6812', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 172726194}]}, 'public_addr': '172.21.15.167:6809/172726194', 'cluster_addr': '172.21.15.167:6811/172726194', 'heartbeat_back_addr': '172.21.15.167:6815/172726194', 'heartbeat_front_addr': '172.21.15.167:6813/172726194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6dad12bf-94de-482a-a402-49e67d5e5dc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6800', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2195746987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6802', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2195746987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6806', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2195746987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6804', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2195746987}]}, 'public_addr': '172.21.15.167:6801/2195746987', 'cluster_addr': '172.21.15.167:6803/2195746987', 'heartbeat_back_addr': '172.21.15.167:6807/2195746987', 'heartbeat_front_addr': '172.21.15.167:6805/2195746987', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e6734cfe-3c24-4ddd-8ef0-5ed450efe772', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6817', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6819', 'nonce': 954791674}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6822', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6825', 'nonce': 954791674}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6830', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6831', 'nonce': 954791674}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6827', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6829', 'nonce': 954791674}]}, 'public_addr': '172.21.15.167:6819/954791674', 'cluster_addr': '172.21.15.167:6825/954791674', 'heartbeat_back_addr': '172.21.15.167:6831/954791674', 'heartbeat_front_addr': '172.21.15.167:6829/954791674', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.509233+0000', 'dead_epoch': 75}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.465696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.405177+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.269202+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.359180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.351834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.490127+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.484842+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.4', 'osds': [5, 7]}, {'pgid': '3.16', 'osds': [3, 2]}, {'pgid': '3.1b', 'osds': [5, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 53, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T00:36:08.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:36:08.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:08.120 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:08.111+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:08.218 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:08.215+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:08.287 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:08.287 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:08.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:08.346 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:08.343+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:08.420 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:08.412+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:08.464 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:08.464 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:08.476 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:36:08.477 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:36:08.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:08.495 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-04-24T00:36:07.730788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1222945941,"num_objects":613,"num_object_clones":0,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":94587,"num_read_kb":758036,"num_write":15661,"num_write_kb":563957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1370,"num_bytes_recovered":89463780,"num_keys_recovered":230,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11372,"ondisk_log_size":11372,"up":82,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":295268,"kb_used_data":47104,"kb_used_omap":9,"kb_used_meta":248054,"kb_avail":565935772,"statfs":{"total":579820584960,"available":579518230528,"internally_reserved":0,"allocated":48234496,"data_stored":356285508,"data_compressed":10139898,"data_compressed_allocated":31551488,"data_compressed_original":348086500,"omap_allocated":9567,"internal_metadata":254007969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70482466,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1208,"num_read_kb":4463,"num_write":282,"num_write_kb":3156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030919"},"pg_stats":[{"pgid":"3.1f","version":"128'671","reported_seq":3707,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702659+0000","last_change":"2024-04-24T00:36:01.695567+0000","last_active":"2024-04-24T00:36:03.702659+0000","last_peered":"2024-04-24T00:36:03.702659+0000","last_clean":"2024-04-24T00:36:03.702659+0000","last_became_active":"2024-04-24T00:34:34.562029+0000","last_became_peered":"2024-04-24T00:34:34.562029+0000","last_unstale":"2024-04-24T00:36:03.702659+0000","last_undegraded":"2024-04-24T00:36:03.702659+0000","last_fullsized":"2024-04-24T00:36:03.702659+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:06.207255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00409465,"stat_sum":{"num_bytes":53678000,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":4803,"num_read_kb":30955,"num_write":1088,"num_write_kb":26919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":107847,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.1e","version":"132'418","reported_seq":2114,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.708298+0000","last_change":"2024-04-24T00:36:01.709497+0000","last_active":"2024-04-24T00:36:03.708298+0000","last_peered":"2024-04-24T00:36:03.708298+0000","last_clean":"2024-04-24T00:36:03.708298+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:36:03.708298+0000","last_undegraded":"2024-04-24T00:36:03.708298+0000","last_fullsized":"2024-04-24T00:36:03.708298+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:58:22.372516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014208700000000001,"stat_sum":{"num_bytes":21004237,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2969,"num_read_kb":7552,"num_write":496,"num_write_kb":10109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.1d","version":"139'223","reported_seq":3713,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702120+0000","last_change":"2024-04-24T00:36:01.696005+0000","last_active":"2024-04-24T00:36:03.702120+0000","last_peered":"2024-04-24T00:36:03.702120+0000","last_clean":"2024-04-24T00:36:03.702120+0000","last_became_active":"2024-04-24T00:34:28.726196+0000","last_became_peered":"2024-04-24T00:34:28.726196+0000","last_unstale":"2024-04-24T00:36:03.702120+0000","last_undegraded":"2024-04-24T00:36:03.702120+0000","last_fullsized":"2024-04-24T00:36:03.702120+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:32:25.750646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043397899999999996,"stat_sum":{"num_bytes":64493490,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4195,"num_read_kb":37662,"num_write":265,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":458933,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.1c","version":"136'496","reported_seq":4215,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.701882+0000","last_change":"2024-04-24T00:36:01.693415+0000","last_active":"2024-04-24T00:36:03.701882+0000","last_peered":"2024-04-24T00:36:03.701882+0000","last_clean":"2024-04-24T00:36:03.701882+0000","last_became_active":"2024-04-24T00:34:31.747052+0000","last_became_peered":"2024-04-24T00:34:31.747052+0000","last_unstale":"2024-04-24T00:36:03.701882+0000","last_undegraded":"2024-04-24T00:36:03.701882+0000","last_fullsized":"2024-04-24T00:36:03.701882+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:55.352254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021834770000000001,"stat_sum":{"num_bytes":48376071,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4301,"num_read_kb":41095,"num_write":627,"num_write_kb":19743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":949407,"num_keys_recovered":27,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.1b","version":"135'239","reported_seq":2679,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.701649+0000","last_change":"2024-04-24T00:36:01.691762+0000","last_active":"2024-04-24T00:36:03.701649+0000","last_peered":"2024-04-24T00:36:03.701649+0000","last_clean":"2024-04-24T00:36:03.701649+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:36:03.701649+0000","last_undegraded":"2024-04-24T00:36:03.701649+0000","last_fullsized":"2024-04-24T00:36:03.701649+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:22:43.823882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014316419999999999,"stat_sum":{"num_bytes":34794362,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2859,"num_read_kb":25848,"num_write":266,"num_write_kb":15108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.1a","version":"138'214","reported_seq":2994,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702831+0000","last_change":"2024-04-24T00:36:01.699117+0000","last_active":"2024-04-24T00:36:03.702831+0000","last_peered":"2024-04-24T00:36:03.702831+0000","last_clean":"2024-04-24T00:36:03.702831+0000","last_became_active":"2024-04-24T00:35:47.582331+0000","last_became_peered":"2024-04-24T00:35:47.582331+0000","last_unstale":"2024-04-24T00:36:03.702831+0000","last_undegraded":"2024-04-24T00:36:03.702831+0000","last_fullsized":"2024-04-24T00:36:03.702831+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:25:51.707980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026079739999999999,"stat_sum":{"num_bytes":9349795,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2432,"num_read_kb":12010,"num_write":258,"num_write_kb":9854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":1449843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.19","version":"135'322","reported_seq":2447,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.708473+0000","last_change":"2024-04-24T00:36:01.702106+0000","last_active":"2024-04-24T00:36:03.708473+0000","last_peered":"2024-04-24T00:36:03.708473+0000","last_clean":"2024-04-24T00:36:03.708473+0000","last_became_active":"2024-04-24T00:35:47.589674+0000","last_became_peered":"2024-04-24T00:35:47.589674+0000","last_unstale":"2024-04-24T00:36:03.708473+0000","last_undegraded":"2024-04-24T00:36:03.708473+0000","last_fullsized":"2024-04-24T00:36:03.708473+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:43.320474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039646700000000001,"stat_sum":{"num_bytes":25652765,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":2241,"num_read_kb":15217,"num_write":526,"num_write_kb":18292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":868041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.18","version":"134'189","reported_seq":2259,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-24T00:36:02.690320+0000","last_change":"2024-04-24T00:36:01.690854+0000","last_active":"2024-04-24T00:36:02.690320+0000","last_peered":"2024-04-24T00:36:02.690320+0000","last_clean":"2024-04-24T00:36:02.690320+0000","last_became_active":"2024-04-24T00:35:47.588569+0000","last_became_peered":"2024-04-24T00:35:47.588569+0000","last_unstale":"2024-04-24T00:36:02.690320+0000","last_undegraded":"2024-04-24T00:36:02.690320+0000","last_fullsized":"2024-04-24T00:36:02.690320+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:23:40.599826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069975500000000004,"stat_sum":{"num_bytes":29396529,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1800,"num_read_kb":20053,"num_write":216,"num_write_kb":19381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":4745782,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.17","version":"138'169","reported_seq":2564,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702349+0000","last_change":"2024-04-24T00:36:01.694314+0000","last_active":"2024-04-24T00:36:03.702349+0000","last_peered":"2024-04-24T00:36:03.702349+0000","last_clean":"2024-04-24T00:36:03.702349+0000","last_became_active":"2024-04-24T00:34:34.561802+0000","last_became_peered":"2024-04-24T00:34:34.561802+0000","last_unstale":"2024-04-24T00:36:03.702349+0000","last_undegraded":"2024-04-24T00:36:03.702349+0000","last_fullsized":"2024-04-24T00:36:03.702349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:48:34.576688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033984179999999998,"stat_sum":{"num_bytes":16689130,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1971,"num_read_kb":19684,"num_write":233,"num_write_kb":9399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":934056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.16","version":"139'468","reported_seq":4282,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.710377+0000","last_change":"2024-04-24T00:36:01.701554+0000","last_active":"2024-04-24T00:36:03.710377+0000","last_peered":"2024-04-24T00:36:03.710377+0000","last_clean":"2024-04-24T00:36:03.710377+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:36:03.710377+0000","last_undegraded":"2024-04-24T00:36:03.710377+0000","last_fullsized":"2024-04-24T00:36:03.710377+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:55:22.729396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045573100000000002,"stat_sum":{"num_bytes":43080969,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5551,"num_read_kb":22133,"num_write":714,"num_write_kb":12680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.15","version":"139'309","reported_seq":4992,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.710322+0000","last_change":"2024-04-24T00:36:01.701648+0000","last_active":"2024-04-24T00:36:03.710322+0000","last_peered":"2024-04-24T00:36:03.710322+0000","last_clean":"2024-04-24T00:36:03.710322+0000","last_became_active":"2024-04-24T00:35:28.725126+0000","last_became_peered":"2024-04-24T00:35:28.725126+0000","last_unstale":"2024-04-24T00:36:03.710322+0000","last_undegraded":"2024-04-24T00:36:03.710322+0000","last_fullsized":"2024-04-24T00:36:03.710322+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:55:06.678503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052360999999999998,"stat_sum":{"num_bytes":59896944,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4563,"num_read_kb":39340,"num_write":400,"num_write_kb":13725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":2312722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.14","version":"139'291","reported_seq":2853,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.703175+0000","last_change":"2024-04-24T00:36:01.700277+0000","last_active":"2024-04-24T00:36:03.703175+0000","last_peered":"2024-04-24T00:36:03.703175+0000","last_clean":"2024-04-24T00:36:03.703175+0000","last_became_active":"2024-04-24T00:35:28.722992+0000","last_became_peered":"2024-04-24T00:35:28.722992+0000","last_unstale":"2024-04-24T00:36:03.703175+0000","last_undegraded":"2024-04-24T00:36:03.703175+0000","last_fullsized":"2024-04-24T00:36:03.703175+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:48.964165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030825539999999999,"stat_sum":{"num_bytes":39936341,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2575,"num_read_kb":21045,"num_write":359,"num_write_kb":20788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1929866,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":257,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.708555+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:36:03.708555+0000","last_peered":"2024-04-24T00:36:03.708555+0000","last_clean":"2024-04-24T00:36:03.708555+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:36:03.708555+0000","last_undegraded":"2024-04-24T00:36:03.708555+0000","last_fullsized":"2024-04-24T00:36:03.708555+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"139'294","reported_seq":4011,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.703307+0000","last_change":"2024-04-24T00:36:01.699905+0000","last_active":"2024-04-24T00:36:03.703307+0000","last_peered":"2024-04-24T00:36:03.703307+0000","last_clean":"2024-04-24T00:36:03.703307+0000","last_became_active":"2024-04-24T00:35:47.582725+0000","last_became_peered":"2024-04-24T00:35:47.582725+0000","last_unstale":"2024-04-24T00:36:03.703307+0000","last_undegraded":"2024-04-24T00:36:03.703307+0000","last_fullsized":"2024-04-24T00:36:03.703307+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:30:58.590068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029382980000000002,"stat_sum":{"num_bytes":58605238,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":3648,"num_read_kb":40777,"num_write":501,"num_write_kb":30879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":11056110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.708403+0000","last_change":"2024-04-24T00:35:44.571029+0000","last_active":"2024-04-24T00:36:03.708403+0000","last_peered":"2024-04-24T00:36:03.708403+0000","last_clean":"2024-04-24T00:36:03.708403+0000","last_became_active":"2024-04-24T00:35:44.570229+0000","last_became_peered":"2024-04-24T00:35:44.570229+0000","last_unstale":"2024-04-24T00:36:03.708403+0000","last_undegraded":"2024-04-24T00:36:03.708403+0000","last_fullsized":"2024-04-24T00:36:03.708403+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:49:14.935258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"139'212","reported_seq":3214,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702226+0000","last_change":"2024-04-24T00:36:01.693575+0000","last_active":"2024-04-24T00:36:03.702226+0000","last_peered":"2024-04-24T00:36:03.702226+0000","last_clean":"2024-04-24T00:36:03.702226+0000","last_became_active":"2024-04-24T00:35:28.721196+0000","last_became_peered":"2024-04-24T00:35:28.721196+0000","last_unstale":"2024-04-24T00:36:03.702226+0000","last_undegraded":"2024-04-24T00:36:03.702226+0000","last_fullsized":"2024-04-24T00:36:03.702226+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:31:33.717365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027790760000000001,"stat_sum":{"num_bytes":45910089,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2728,"num_read_kb":27145,"num_write":255,"num_write_kb":14688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4076209,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":176,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702860+0000","last_change":"2024-04-24T00:35:44.571723+0000","last_active":"2024-04-24T00:36:03.702860+0000","last_peered":"2024-04-24T00:36:03.702860+0000","last_clean":"2024-04-24T00:36:03.702860+0000","last_became_active":"2024-04-24T00:35:44.570858+0000","last_became_peered":"2024-04-24T00:35:44.570858+0000","last_unstale":"2024-04-24T00:36:03.702860+0000","last_undegraded":"2024-04-24T00:36:03.702860+0000","last_fullsized":"2024-04-24T00:36:03.702860+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:16.447144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"119'343","reported_seq":2243,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-24T00:36:02.690701+0000","last_change":"2024-04-24T00:36:01.691360+0000","last_active":"2024-04-24T00:36:02.690701+0000","last_peered":"2024-04-24T00:36:02.690701+0000","last_clean":"2024-04-24T00:36:02.690701+0000","last_became_active":"2024-04-24T00:35:47.586287+0000","last_became_peered":"2024-04-24T00:35:47.586287+0000","last_unstale":"2024-04-24T00:36:02.690701+0000","last_undegraded":"2024-04-24T00:36:02.690701+0000","last_fullsized":"2024-04-24T00:36:02.690701+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:07.066115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054344400000000002,"stat_sum":{"num_bytes":23735278,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":2226,"num_read_kb":13706,"num_write":524,"num_write_kb":10049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1579365,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-24T00:36:02.690474+0000","last_change":"2024-04-24T00:35:50.611737+0000","last_active":"2024-04-24T00:36:02.690474+0000","last_peered":"2024-04-24T00:36:02.690474+0000","last_clean":"2024-04-24T00:36:02.690474+0000","last_became_active":"2024-04-24T00:35:50.611171+0000","last_became_peered":"2024-04-24T00:35:50.611171+0000","last_unstale":"2024-04-24T00:36:02.690474+0000","last_undegraded":"2024-04-24T00:36:02.690474+0000","last_fullsized":"2024-04-24T00:36:02.690474+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:02:25.937574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"139'276","reported_seq":1777,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702031+0000","last_change":"2024-04-24T00:36:01.694565+0000","last_active":"2024-04-24T00:36:03.702031+0000","last_peered":"2024-04-24T00:36:03.702031+0000","last_clean":"2024-04-24T00:36:03.702031+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:36:03.702031+0000","last_undegraded":"2024-04-24T00:36:03.702031+0000","last_fullsized":"2024-04-24T00:36:03.702031+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:47:05.286843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034578510000000001,"stat_sum":{"num_bytes":35871812,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1688,"num_read_kb":16683,"num_write":341,"num_write_kb":15712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.3","version":"135'214","reported_seq":1940,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.703095+0000","last_change":"2024-04-24T00:36:01.702567+0000","last_active":"2024-04-24T00:36:03.703095+0000","last_peered":"2024-04-24T00:36:03.703095+0000","last_clean":"2024-04-24T00:36:03.703095+0000","last_became_active":"2024-04-24T00:34:34.549707+0000","last_became_peered":"2024-04-24T00:34:34.549707+0000","last_unstale":"2024-04-24T00:36:03.703095+0000","last_undegraded":"2024-04-24T00:36:03.703095+0000","last_fullsized":"2024-04-24T00:36:03.703095+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:08.706856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047223320000000001,"stat_sum":{"num_bytes":34876671,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1665,"num_read_kb":22494,"num_write":204,"num_write_kb":14605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33262,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":182,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.701580+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:36:03.701580+0000","last_peered":"2024-04-24T00:36:03.701580+0000","last_clean":"2024-04-24T00:36:03.701580+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:36:03.701580+0000","last_undegraded":"2024-04-24T00:36:03.701580+0000","last_fullsized":"2024-04-24T00:36:03.701580+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"139'340","reported_seq":2596,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-24T00:36:02.690478+0000","last_change":"2024-04-24T00:36:01.690712+0000","last_active":"2024-04-24T00:36:02.690478+0000","last_peered":"2024-04-24T00:36:02.690478+0000","last_clean":"2024-04-24T00:36:02.690478+0000","last_became_active":"2024-04-24T00:35:47.586354+0000","last_became_peered":"2024-04-24T00:35:47.586354+0000","last_unstale":"2024-04-24T00:36:02.690478+0000","last_undegraded":"2024-04-24T00:36:02.690478+0000","last_fullsized":"2024-04-24T00:36:02.690478+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:01.610229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031859,"stat_sum":{"num_bytes":26918976,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2867,"num_read_kb":23449,"num_write":317,"num_write_kb":12067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3429938,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":250,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-24T00:36:02.690531+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:36:02.690531+0000","last_peered":"2024-04-24T00:36:02.690531+0000","last_clean":"2024-04-24T00:36:02.690531+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:36:02.690531+0000","last_undegraded":"2024-04-24T00:36:02.690531+0000","last_fullsized":"2024-04-24T00:36:02.690531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'209","reported_seq":2010,"reported_epoch":140,"state":"remapped+peering","last_fresh":"2024-04-24T00:36:03.702278+0000","last_change":"2024-04-24T00:36:03.702278+0000","last_active":"2024-04-24T00:36:02.697680+0000","last_peered":"2024-04-24T00:36:02.504522+0000","last_clean":"2024-04-24T00:36:02.504522+0000","last_became_active":"2024-04-24T00:35:53.626193+0000","last_became_peered":"2024-04-24T00:35:53.626193+0000","last_unstale":"2024-04-24T00:36:03.702278+0000","last_undegraded":"2024-04-24T00:36:03.702278+0000","last_fullsized":"2024-04-24T00:36:03.702278+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013209700000000001,"stat_sum":{"num_bytes":51816170,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1260,"num_read_kb":17027,"num_write":261,"num_write_kb":14735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":6028885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.708955+0000","last_change":"2024-04-24T00:35:44.570143+0000","last_active":"2024-04-24T00:36:03.708955+0000","last_peered":"2024-04-24T00:36:03.708955+0000","last_clean":"2024-04-24T00:36:03.708955+0000","last_became_active":"2024-04-24T00:35:44.569488+0000","last_became_peered":"2024-04-24T00:35:44.569488+0000","last_unstale":"2024-04-24T00:36:03.708955+0000","last_undegraded":"2024-04-24T00:36:03.708955+0000","last_fullsized":"2024-04-24T00:36:03.708955+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:35:29.076753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"134'338","reported_seq":1086,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.701691+0000","last_change":"2024-04-24T00:36:01.693441+0000","last_active":"2024-04-24T00:36:03.701691+0000","last_peered":"2024-04-24T00:36:03.701691+0000","last_clean":"2024-04-24T00:36:03.701691+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:36:03.701691+0000","last_undegraded":"2024-04-24T00:36:03.701691+0000","last_fullsized":"2024-04-24T00:36:03.701691+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:16:07.225176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025226670000000001,"stat_sum":{"num_bytes":40041619,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1574,"num_read_kb":11004,"num_write":472,"num_write_kb":20019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":182,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.701504+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:36:03.701504+0000","last_peered":"2024-04-24T00:36:03.701504+0000","last_clean":"2024-04-24T00:36:03.701504+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:36:03.701504+0000","last_undegraded":"2024-04-24T00:36:03.701504+0000","last_fullsized":"2024-04-24T00:36:03.701504+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":349,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.708026+0000","last_change":"2024-04-24T00:35:47.589387+0000","last_active":"2024-04-24T00:36:03.708026+0000","last_peered":"2024-04-24T00:36:03.708026+0000","last_clean":"2024-04-24T00:36:03.708026+0000","last_became_active":"2024-04-24T00:35:47.588827+0000","last_became_peered":"2024-04-24T00:35:47.588827+0000","last_unstale":"2024-04-24T00:36:03.708026+0000","last_undegraded":"2024-04-24T00:36:03.708026+0000","last_fullsized":"2024-04-24T00:36:03.708026+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:09:59.244674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"135'656","reported_seq":3172,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702378+0000","last_change":"2024-04-24T00:36:01.698111+0000","last_active":"2024-04-24T00:36:03.702378+0000","last_peered":"2024-04-24T00:36:03.702378+0000","last_clean":"2024-04-24T00:36:03.702378+0000","last_became_active":"2024-04-24T00:35:54.570587+0000","last_became_peered":"2024-04-24T00:35:54.570587+0000","last_unstale":"2024-04-24T00:36:03.702378+0000","last_undegraded":"2024-04-24T00:36:03.702378+0000","last_fullsized":"2024-04-24T00:36:03.702378+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:00:58.106852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029736070000000001,"stat_sum":{"num_bytes":48362638,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4363,"num_read_kb":27028,"num_write":916,"num_write_kb":17395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":5566512,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.9","version":"138'476","reported_seq":3204,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.703360+0000","last_change":"2024-04-24T00:36:01.698853+0000","last_active":"2024-04-24T00:36:03.703360+0000","last_peered":"2024-04-24T00:36:03.703360+0000","last_clean":"2024-04-24T00:36:03.703360+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:36:03.703360+0000","last_undegraded":"2024-04-24T00:36:03.703360+0000","last_fullsized":"2024-04-24T00:36:03.703360+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:22.671451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026016780000000001,"stat_sum":{"num_bytes":46291498,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4757,"num_read_kb":20541,"num_write":575,"num_write_kb":15348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.a","version":"139'543","reported_seq":1913,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.708126+0000","last_change":"2024-04-24T00:36:01.710798+0000","last_active":"2024-04-24T00:36:03.708126+0000","last_peered":"2024-04-24T00:36:03.708126+0000","last_clean":"2024-04-24T00:36:03.708126+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:36:03.708126+0000","last_undegraded":"2024-04-24T00:36:03.708126+0000","last_fullsized":"2024-04-24T00:36:03.708126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:57:48.610785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029627339999999999,"stat_sum":{"num_bytes":45608664,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2788,"num_read_kb":28370,"num_write":861,"num_write_kb":24935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.b","version":"137'159","reported_seq":1210,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702800+0000","last_change":"2024-04-24T00:36:01.699621+0000","last_active":"2024-04-24T00:36:03.702800+0000","last_peered":"2024-04-24T00:36:03.702800+0000","last_clean":"2024-04-24T00:36:03.702800+0000","last_became_active":"2024-04-24T00:34:34.549802+0000","last_became_peered":"2024-04-24T00:34:34.549802+0000","last_unstale":"2024-04-24T00:36:03.702800+0000","last_undegraded":"2024-04-24T00:36:03.702800+0000","last_fullsized":"2024-04-24T00:36:03.702800+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:22:15.479937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030462940000000002,"stat_sum":{"num_bytes":11945415,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1273,"num_read_kb":10791,"num_write":171,"num_write_kb":6460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":722108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.c","version":"116'507","reported_seq":5044,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.701881+0000","last_change":"2024-04-24T00:36:01.695332+0000","last_active":"2024-04-24T00:36:03.701881+0000","last_peered":"2024-04-24T00:36:03.701881+0000","last_clean":"2024-04-24T00:36:03.701881+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:36:03.701881+0000","last_undegraded":"2024-04-24T00:36:03.701881+0000","last_fullsized":"2024-04-24T00:36:03.701881+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:23:44.879653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036761820000000001,"stat_sum":{"num_bytes":63125637,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5817,"num_read_kb":36070,"num_write":790,"num_write_kb":20815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.d","version":"139'164","reported_seq":901,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.701966+0000","last_change":"2024-04-24T00:36:01.692448+0000","last_active":"2024-04-24T00:36:03.701966+0000","last_peered":"2024-04-24T00:36:03.701966+0000","last_clean":"2024-04-24T00:36:03.701966+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:36:03.701966+0000","last_undegraded":"2024-04-24T00:36:03.701966+0000","last_fullsized":"2024-04-24T00:36:03.701966+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:52:53.778949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019627889999999999,"stat_sum":{"num_bytes":21697463,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":414,"num_read_kb":14642,"num_write":191,"num_write_kb":15079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.e","version":"136'327","reported_seq":1736,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.703450+0000","last_change":"2024-04-24T00:36:01.701897+0000","last_active":"2024-04-24T00:36:03.703450+0000","last_peered":"2024-04-24T00:36:03.703450+0000","last_clean":"2024-04-24T00:36:03.703450+0000","last_became_active":"2024-04-24T00:35:29.560215+0000","last_became_peered":"2024-04-24T00:35:29.560215+0000","last_unstale":"2024-04-24T00:36:03.703450+0000","last_undegraded":"2024-04-24T00:36:03.703450+0000","last_fullsized":"2024-04-24T00:36:03.703450+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:06:55.917657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043804079999999997,"stat_sum":{"num_bytes":32631587,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1594,"num_read_kb":34103,"num_write":566,"num_write_kb":25626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":2728103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.f","version":"138'317","reported_seq":2944,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.703061+0000","last_change":"2024-04-24T00:36:01.701410+0000","last_active":"2024-04-24T00:36:03.703061+0000","last_peered":"2024-04-24T00:36:03.703061+0000","last_clean":"2024-04-24T00:36:03.703061+0000","last_became_active":"2024-04-24T00:35:47.587236+0000","last_became_peered":"2024-04-24T00:35:47.587236+0000","last_unstale":"2024-04-24T00:36:03.703061+0000","last_undegraded":"2024-04-24T00:36:03.703061+0000","last_fullsized":"2024-04-24T00:36:03.703061+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:39:46.504773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003996543,"stat_sum":{"num_bytes":30576704,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2586,"num_read_kb":20114,"num_write":415,"num_write_kb":24745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":6390656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.10","version":"135'317","reported_seq":1670,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.708113+0000","last_change":"2024-04-24T00:36:01.708805+0000","last_active":"2024-04-24T00:36:03.708113+0000","last_peered":"2024-04-24T00:36:03.708113+0000","last_clean":"2024-04-24T00:36:03.708113+0000","last_became_active":"2024-04-24T00:34:34.557992+0000","last_became_peered":"2024-04-24T00:34:34.557992+0000","last_unstale":"2024-04-24T00:36:03.708113+0000","last_undegraded":"2024-04-24T00:36:03.708113+0000","last_fullsized":"2024-04-24T00:36:03.708113+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:42.349254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082122100000000002,"stat_sum":{"num_bytes":12046753,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2240,"num_read_kb":7462,"num_write":392,"num_write_kb":7347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":179074,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.11","version":"135'378","reported_seq":2439,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702529+0000","last_change":"2024-04-24T00:36:01.698231+0000","last_active":"2024-04-24T00:36:03.702529+0000","last_peered":"2024-04-24T00:36:03.702529+0000","last_clean":"2024-04-24T00:36:03.702529+0000","last_became_active":"2024-04-24T00:35:28.722568+0000","last_became_peered":"2024-04-24T00:35:28.722568+0000","last_unstale":"2024-04-24T00:36:03.702529+0000","last_undegraded":"2024-04-24T00:36:03.702529+0000","last_fullsized":"2024-04-24T00:36:03.702529+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:35:42.087925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002835752,"stat_sum":{"num_bytes":72288358,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":2198,"num_read_kb":46038,"num_write":433,"num_write_kb":33772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":1618865,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.12","version":"134'992","reported_seq":3714,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-24T00:36:02.690458+0000","last_change":"2024-04-24T00:36:01.691023+0000","last_active":"2024-04-24T00:36:02.690458+0000","last_peered":"2024-04-24T00:36:02.690458+0000","last_clean":"2024-04-24T00:36:02.690458+0000","last_became_active":"2024-04-24T00:35:53.623520+0000","last_became_peered":"2024-04-24T00:35:53.623520+0000","last_unstale":"2024-04-24T00:36:02.690458+0000","last_undegraded":"2024-04-24T00:36:02.690458+0000","last_fullsized":"2024-04-24T00:36:02.690458+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":992,"log_dups_size":0,"ondisk_log_size":992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:23.573825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057313899999999996,"stat_sum":{"num_bytes":41640823,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5810,"num_read_kb":32315,"num_write":1653,"num_write_kb":29411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":19625123,"num_keys_recovered":43,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]},{"pgid":"3.13","version":"139'184","reported_seq":3114,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-24T00:36:03.702950+0000","last_change":"2024-04-24T00:36:01.700706+0000","last_active":"2024-04-24T00:36:03.702950+0000","last_peered":"2024-04-24T00:36:03.702950+0000","last_clean":"2024-04-24T00:36:03.702950+0000","last_became_active":"2024-04-24T00:35:28.722941+0000","last_became_peered":"2024-04-24T00:35:28.722941+0000","last_unstale":"2024-04-24T00:36:03.702950+0000","last_undegraded":"2024-04-24T00:36:03.702950+0000","last_fullsized":"2024-04-24T00:36:03.702950+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:44:21.489730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034756539999999999,"stat_sum":{"num_bytes":32015547,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2715,"num_read_kb":15583,"num_write":216,"num_write_kb":17550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":962033,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1222355573,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":94471,"num_read_kb":757936,"num_write":15502,"num_write_kb":561541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":88283044,"num_keys_recovered":230,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43065344,"data_stored":382589695,"data_compressed":11843584,"data_compressed_allocated":35188736,"data_compressed_original":378577318,"omap_allocated":0,"internal_metadata":0},"log_size":11255,"ondisk_log_size":11255,"up":64,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":2951840,"data_compressed":33203,"data_compressed_allocated":184320,"data_compressed_original":2916352,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46868,"kb_used_data":7360,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94324972,"statfs":{"total":96636764160,"available":96588771328,"internally_reserved":0,"allocated":7536640,"data_stored":63800438,"data_compressed":1110643,"data_compressed_allocated":4972544,"data_compressed_original":62563669,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38832,"kb_used_data":11804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333008,"statfs":{"total":96636764160,"available":96597000192,"internally_reserved":0,"allocated":12087296,"data_stored":89750798,"data_compressed":3119343,"data_compressed_allocated":8519680,"data_compressed_original":87974438,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61884,"kb_used_data":12656,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94309956,"statfs":{"total":96636764160,"available":96573394944,"internally_reserved":0,"allocated":12959744,"data_stored":113339668,"data_compressed":2661191,"data_compressed_allocated":9506816,"data_compressed_original":111635092,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56148,"kb_used_data":6916,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94315692,"statfs":{"total":96636764160,"available":96579268608,"internally_reserved":0,"allocated":7081984,"data_stored":42132110,"data_compressed":1778541,"data_compressed_allocated":4304896,"data_compressed_original":40760548,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45252,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94326588,"statfs":{"total":96636764160,"available":96590426112,"internally_reserved":0,"allocated":5890048,"data_stored":33420794,"data_compressed":1348884,"data_compressed_allocated":3321856,"data_compressed_original":32156145,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46284,"kb_used_data":2616,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325556,"statfs":{"total":96636764160,"available":96589369344,"internally_reserved":0,"allocated":2678784,"data_stored":13841700,"data_compressed":121296,"data_compressed_allocated":925696,"data_compressed_original":12996608,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":408021893131,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":7369185,"data_compressed":431142,"data_compressed_allocated":892928,"data_compressed_original":7188480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1011712,"data_stored":13035912,"data_compressed":121296,"data_compressed_allocated":925696,"data_compressed_original":12996608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4182016,"data_stored":32024638,"data_compressed":1341899,"data_compressed_allocated":3284992,"data_compressed_original":31566321,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5414912,"data_stored":41326322,"data_compressed":1778541,"data_compressed_allocated":4304896,"data_compressed_original":40760548,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11292672,"data_stored":112533880,"data_compressed":2661191,"data_compressed_allocated":9506816,"data_compressed_original":111635092,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10420224,"data_stored":88945010,"data_compressed":3119343,"data_compressed_allocated":8519680,"data_compressed_original":87974438,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5828608,"data_stored":62404282,"data_compressed":1103658,"data_compressed_allocated":4935680,"data_compressed_original":61973845,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:36:08.497 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [7, 3] 2024-04-24T00:36:08.497 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.3', '7', '3'] 2024-04-24T00:36:08.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.3 7 3 2024-04-24T00:36:08.538 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:08.532+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:08.596 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:08.597 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:08.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:08.658 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:08.655+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:08.723 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:08.719+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:08.737 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:08.737 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:08.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:08.824 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:08.816+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:08.925 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:08.923+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:08.944 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:08.944 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:08.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:09.050 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:09.047+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:09.120 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:09.121 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:09.128 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:09.124+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:09.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:09.228 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:09.220+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:09.302 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:09.302 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:09.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:09.329 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:09.327+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:09.430 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:09.424+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:09.452 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:09.452 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:09.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:09.532 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:09.527+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:09.633 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:09.627+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:09.639 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:09.639 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:09.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:09.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:09.733 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:09.731+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:09.752 INFO:teuthology.orchestra.run.smithi064.stderr:set 2.3 pg_upmap mapping to [7,3] 2024-04-24T00:36:09.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:36:09.820 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:09.820 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:09.835 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:09.831+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:09.838 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:09.838 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:09.846 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:09.847 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-04-24T00:36:09.847 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:09.855 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:09.855 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:09.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:09.873 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:09.873 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:09.895 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:09.918 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:09.919 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:09.929 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-04-24T00:36:09.936 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:09.931+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:09.940 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:09.940 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:09.961 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:09.962 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:09.992 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:09.992 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:09.992 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:09.992 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:10.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:10.057 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:10.052+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:10.137 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:10.135+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:10.184 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:10.185 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:10.185 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:10.185 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:10.185 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:10.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:10.238 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:10.235+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:10.321 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 102.883037495, 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.322 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:10.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:36:10.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 102.827526009, 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:36:10.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.726949360999996, 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:36:10.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.685384407000001, 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.338 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:36:10.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:36:10.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:36:10.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.121652845, 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.342 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:36:10.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:36:10.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.618679448000002, 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:36:10.353 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.380365214999998, 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:36:10.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:36:10.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:36:10.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:36:10.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345424792000003, 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.360 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:36:10.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:36:10.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:36:10.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345403918999999, 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:10.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.368 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345380540000001, 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:10.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345358998000002, 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:36:10.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345337477999998, 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:36:10.381 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345291320999998, 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.382 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:36:10.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345267202000002, 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:36:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345243797999998, 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:36:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:36:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345216024999999, 2024-04-24T00:36:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:36:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345194612, 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:36:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:36:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:36:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.345174020999998, 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:36:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.129103779000001, 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:36:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.101132875000001, 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:36:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:36:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:36:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.418 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:10.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:10.421 INFO:teuthology.orchestra.run.smithi064.stdout:3961 punch from 0x26446ac to 0x264d61a, (0x8f6e bytes) 2024-04-24T00:36:10.421 INFO:teuthology.orchestra.run.smithi064.stdout:3962 write 0x5bdd2c thru 0x5c718b (0x9460 bytes) 2024-04-24T00:36:10.421 INFO:teuthology.orchestra.run.smithi064.stdout:3963 punch from 0x7f02f60 to 0x7f0b2af, (0x834f bytes) 2024-04-24T00:36:10.421 INFO:teuthology.orchestra.run.smithi064.stdout:3964 read 0x2b30096 thru 0x2b36a70 (0x69db bytes) 2024-04-24T00:36:10.421 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:10.421 INFO:teuthology.orchestra.run.smithi064.stdout:3966 write 0x15a3a2e thru 0x15b26d6 (0xeca9 bytes) 2024-04-24T00:36:10.421 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:10.422 INFO:teuthology.orchestra.run.smithi064.stdout:3970 write 0x8378608 thru 0x838506c (0xca65 bytes) 2024-04-24T00:36:10.422 INFO:teuthology.orchestra.run.smithi064.stdout:3971 trunc from 0x96588ef to 0xe0f81ef 2024-04-24T00:36:10.422 INFO:teuthology.orchestra.run.smithi064.stdout:3972 punch from 0x6ca65d9 to 0x6ca6e5d, (0x884 bytes) 2024-04-24T00:36:10.422 INFO:teuthology.orchestra.run.smithi064.stdout:3973 write 0x95fc2b6 thru 0x9606d41 (0xaa8c bytes) 2024-04-24T00:36:10.422 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:10.422 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:10.412+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:10.426 INFO:teuthology.orchestra.run.smithi064.stdout:3975 trunc from 0xe0f81ef to 0x14fc57e 2024-04-24T00:36:10.483 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:10.488 INFO:teuthology.orchestra.run.smithi064.stdout:3977 trunc from 0x14fc57e to 0x4cd32d0 2024-04-24T00:36:10.502 INFO:teuthology.orchestra.run.smithi064.stdout:3978 read 0x2a09aa7 thru 0x2a16034 (0xc58e bytes) 2024-04-24T00:36:10.520 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:10.515+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:10.525 INFO:teuthology.orchestra.run.smithi064.stdout:3979 write 0xbdc9265 thru 0xbdd7813 (0xe5af bytes) 2024-04-24T00:36:10.533 INFO:teuthology.orchestra.run.smithi064.stdout:3983 writesame 0x45ad060 thru 0x45b38ca data_size 0x1(0x686b bytes) 2024-04-24T00:36:10.548 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:10.549 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:10.549 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:10.549 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:10.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:10.621 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:10.612+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:10.633 INFO:teuthology.orchestra.run.smithi064.stdout:3985 writesame 0x5d759a5 thru 0x5d7bccf data_size 0x1(0x632b bytes) 2024-04-24T00:36:10.682 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:10.682 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:10.682 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:10.682 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:10.682 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:10.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:10.734 INFO:teuthology.orchestra.run.smithi064.stdout:3987 write 0xade5799 thru 0xadf35d9 (0xde41 bytes) 2024-04-24T00:36:10.735 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:10.728+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:10.750 INFO:teuthology.orchestra.run.smithi064.stdout:3988 write 0xbc9a35c thru 0xbca43cd (0xa072 bytes) 2024-04-24T00:36:10.758 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:36:10.764 INFO:teuthology.orchestra.run.smithi064.stdout:3989 read 0xb561f48 thru 0xb56de53 (0xbf0c bytes) 2024-04-24T00:36:10.765 INFO:teuthology.orchestra.run.smithi064.stdout:3992 writesame 0x89ca29f thru 0x89cd37a data_size 0x4(0x30dc bytes) 2024-04-24T00:36:10.776 INFO:teuthology.orchestra.run.smithi064.stdout:3993 trunc from 0xbdd7814 to 0xe5624ab 2024-04-24T00:36:10.789 INFO:teuthology.orchestra.run.smithi064.stdout:3994 punch from 0x1e22102 to 0x1e22a05, (0x903 bytes) 2024-04-24T00:36:10.789 INFO:teuthology.orchestra.run.smithi064.stdout:3995 punch from 0xe52935b to 0xe52fcf5, (0x699a bytes) 2024-04-24T00:36:10.797 INFO:teuthology.orchestra.run.smithi064.stdout:3996 trunc from 0xe5624ab to 0x7e722a5 2024-04-24T00:36:10.823 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:10.819+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:10.833 INFO:teuthology.orchestra.run.smithi064.stdout:3997 trunc from 0x7e722a5 to 0x1a16395 2024-04-24T00:36:10.858 INFO:teuthology.orchestra.run.smithi064.stdout:4000 read 0x126705f thru 0x1270065 (0x9007 bytes) 2024-04-24T00:36:10.872 INFO:teuthology.orchestra.run.smithi064.stdout:4001 write 0x292bf20 thru 0x2937f98 (0xc079 bytes) 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.288181213, 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:36:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.591486429, 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:36:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:36:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.785954469999993, 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:77813 3.0 e102/21 3:05afdb1e:::rbd_data.115e805f80b3.0000000000000004:head v 102'237, mlcod=102'235)", 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.206514120999998, 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074761180000000003, 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664825+0000", 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664911+0000", 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5873999999999998e-05 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664944+0000", 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664959+0000", 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5979000000000001e-05 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665112+0000", 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152943 2024-04-24T00:36:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665157+0000", 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4725000000000003e-05 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672178+0000", 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070206390000000004 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672298+0000", 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120308 2024-04-24T00:36:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head, version: 45'133, data_included: [0~131072,196158~37038,262144~131072,524288~131072,786432~262144,1258308~52412,1441792~131072,3014656~131072,3267061~6628,6160384~34994], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head@45'133, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 6195378, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.949275038000003, 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011957290000000001, 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922063+0000", 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.306e-06 2024-04-24T00:36:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923213+0000", 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011495310000000001 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923258+0000", 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4747999999999999e-05 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923284+0000", 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5873000000000001e-05 2024-04-24T00:36:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923436+0000", 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015226500000000001 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923452+0000", 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6022e-05 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.934018+0000", 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010566545 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head, version: 45'137, data_included: [0~131072,262144~131072,524288~131072,786432~199751,1179648~1552,1441792~3851,3014656~131072,5775043~64169,6160384~131072], data_size: 924683, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head@45'137, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8781824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.920450316, 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011581296, 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950888+0000", 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951161+0000", 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027280499999999999 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951196+0000", 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123999999999998e-05 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951224+0000", 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8045e-05 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951356+0000", 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013187799999999999 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951376+0000", 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.003e-05 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.962467+0000", 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011091474 2024-04-24T00:36:10.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.715115501, 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head, version: 80'319, data_included: [0~393216,524288~131072,1179648~524288], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head@80'319, size: 13631488, copy_subset: [(0, 13631488)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.702361408000002, 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089307889999999997, 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:10.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168976+0000", 2024-04-24T00:36:10.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169620+0000", 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064358299999999998 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169650+0000", 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0111e-05 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169671+0000", 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1231000000000001e-05 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169789+0000", 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118368 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169803+0000", 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3779e-05 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.177905+0000", 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081024800000000004 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.690214462, 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:10.913 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.673470303999999, 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.583321310999999, 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:10.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:10.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.176164631999999, 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:10.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.165777935000001, 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:10.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:10.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.156834646, 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:10.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:10.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:10.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:90104 3.18 3:193b8512:::rbd_data.115e56aa02fc.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write 1644318~44706] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.760701752999999, 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084175689999999997, 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 90104 2024-04-24T00:36:10.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110636+0000", 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:36:10.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110706+0000", 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9392000000000002e-05 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110734+0000", 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8538999999999999e-05 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110747+0000", 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860000000000001e-05 2024-04-24T00:36:10.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110850+0000", 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102988 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111107+0000", 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025716600000000002 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111346+0000", 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000238373 2024-04-24T00:36:10.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.117684+0000", 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063379949999999999 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118724+0000", 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001040685 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118914+0000", 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018984699999999999 2024-04-24T00:36:10.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.119052+0000", 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137701 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPushReply(3.12 128/127 [PushReplyOp(3:4e045fb2:::rbd_data.115e6f07caf6.0000000000000007:head)])", 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.493512383999999, 2024-04-24T00:36:10.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089128709999999993, 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377827+0000", 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5870000000000001e-06 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377852+0000", 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5083e-05 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377890+0000", 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7279e-05 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377914+0000", 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4496000000000001e-05 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.378154+0000", 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024028 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.386737+0000", 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085821460000000006 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91841 3.0 3:02b978df:::rbd_data.115ee57427d8.0000000000000008:head [write 6244202~37028] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.590049+0000", 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.2812868420000001, 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070732210000000002, 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91841 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590049+0000", 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590049+0000", 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590052+0000", 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5940000000000001e-06 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590085+0000", 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2904999999999998e-05 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590108+0000", 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3649999999999999e-05 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590120+0000", 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1909999999999999e-05 2024-04-24T00:36:10.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590285+0000", 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016510599999999999 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.590572+0000", 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028664700000000001 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:10.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.591013+0000", 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044140999999999999 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.596705+0000", 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056917160000000003 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.597034+0000", 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000329191 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.597080+0000", 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5484e-05 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.597122+0000", 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2608000000000003e-05 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:92353 3.0 3:02b978df:::rbd_data.115ee57427d8.0000000000000008:head [truncate 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:10.950 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1178720960000001, 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091800189999999993, 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 92353 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753467+0000", 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7259999999999998e-06 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753478+0000", 2024-04-24T00:36:10.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1456e-05 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753503+0000", 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5015999999999999e-05 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753516+0000", 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.259e-05 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753615+0000", 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9104999999999994e-05 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753624+0000", 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9309999999999997e-06 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755534+0000", 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019101700000000001 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755620+0000", 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5736999999999996e-05 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755756+0000", 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013602800000000001 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.756642+0000", 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088663200000000005 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762416+0000", 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057733849999999998 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762534+0000", 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118594 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762644+0000", 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010964900000000001 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:94309 3.0 3:06002a2d:::rbd_data.115eb3626b1c.000000000000002f:head [delete] snapc 37=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:10.447261+0000", 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.42407558899999998, 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075631839999999997, 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 94309 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447261+0000", 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447261+0000", 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447262+0000", 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.553e-06 2024-04-24T00:36:10.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447268+0000", 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4559999999999999e-06 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447282+0000", 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4763000000000001e-05 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447289+0000", 2024-04-24T00:36:10.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1720000000000001e-06 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447337+0000", 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7184000000000001e-05 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447396+0000", 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9614999999999998e-05 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447458+0000", 2024-04-24T00:36:10.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1284000000000005e-05 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447968+0000", 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000510562 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.454763+0000", 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067951620000000004 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.454793+0000", 2024-04-24T00:36:10.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9448999999999999e-05 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.454824+0000", 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0984e-05 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:94358 3.0 3:06002a2d:::rbd_data.115eb3626b1c.000000000000002f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2512793~56897] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.13227228399999999, 2024-04-24T00:36:10.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010252809999999999, 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 94358 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739067+0000", 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9299999999999999e-06 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739156+0000", 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9148999999999996e-05 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739189+0000", 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2772999999999999e-05 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739205+0000", 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6722999999999999e-05 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739285+0000", 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9729999999999997e-05 2024-04-24T00:36:10.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739348+0000", 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3309e-05 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739445+0000", 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6378000000000005e-05 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.742520+0000", 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030747819999999999 2024-04-24T00:36:10.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749133+0000", 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066130659999999999 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749238+0000", 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010506299999999999 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749317+0000", 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8906999999999995e-05 2024-04-24T00:36:10.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:10.968 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:10.968 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:10.969 INFO:teuthology.orchestra.run.smithi064.stdout:4005 trunc from 0x2937f99 to 0x89c24d5 2024-04-24T00:36:10.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi064.stdout:4007 write 0x297a996 thru 0x297aea2 (0x50d bytes) 2024-04-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi064.stdout:4008 read 0x3ded55c thru 0x3df130d (0x3db2 bytes) 2024-04-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi064.stdout:4010 write 0xaaa474 thru 0xaac3e6 (0x1f73 bytes) 2024-04-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi064.stdout:4011 trunc from 0x89c24d5 to 0xa011deb 2024-04-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi064.stdout:4013 read 0x6290ca9 thru 0x629fd3a (0xf092 bytes) 2024-04-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi064.stdout:4015 trunc from 0xa011deb to 0xd6f6e 2024-04-24T00:36:10.971 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:10.964+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:10.989 INFO:teuthology.orchestra.run.smithi064.stdout:4016 read 0x72ef thru 0x1520e (0xdf20 bytes) 2024-04-24T00:36:11.032 INFO:teuthology.orchestra.run.smithi064.stdout:4017 write 0xc1c550d thru 0xc1c7723 (0x2217 bytes) 2024-04-24T00:36:11.039 INFO:teuthology.orchestra.run.smithi064.stdout:4018 write 0x8532166 thru 0x853282c (0x6c7 bytes) 2024-04-24T00:36:11.046 INFO:teuthology.orchestra.run.smithi064.stdout:4019 read 0xa14406e thru 0xa14c67c (0x860f bytes) 2024-04-24T00:36:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:4020 write 0xd4a8125 thru 0xd4ac920 (0x47fc bytes) 2024-04-24T00:36:11.069 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:11.064+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:11.071 INFO:teuthology.orchestra.run.smithi064.stdout:4021 read 0x74a4e17 thru 0x74aee3f (0xa029 bytes) 2024-04-24T00:36:11.073 INFO:teuthology.orchestra.run.smithi064.stdout:4022 punch from 0x1e67ba8 to 0x1e68539, (0x991 bytes) 2024-04-24T00:36:11.073 INFO:teuthology.orchestra.run.smithi064.stdout:4023 write 0x91deea7 thru 0x91e6914 (0x7a6e bytes) 2024-04-24T00:36:11.086 INFO:teuthology.orchestra.run.smithi064.stdout:4024 write 0x57bd2fe thru 0x57c801e (0xad21 bytes) 2024-04-24T00:36:11.098 INFO:teuthology.orchestra.run.smithi064.stdout:4025 write 0x72dd0db thru 0x72dd9ec (0x912 bytes) 2024-04-24T00:36:11.108 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:11.109 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:11.128 INFO:teuthology.orchestra.run.smithi064.stdout:4026 trunc from 0xd4ac921 to 0x1614fa0 2024-04-24T00:36:11.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:11.170 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:11.167+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:11.190 INFO:teuthology.orchestra.run.smithi064.stdout:4027 trunc from 0x1614fa0 to 0xc5c02d1 2024-04-24T00:36:11.204 INFO:teuthology.orchestra.run.smithi064.stdout:4029 read 0x528d53 thru 0x536bec (0xde9a bytes) 2024-04-24T00:36:11.219 INFO:teuthology.orchestra.run.smithi064.stdout:4030 write 0xd389c3a thru 0xd390dba (0x7181 bytes) 2024-04-24T00:36:11.226 INFO:teuthology.orchestra.run.smithi064.stdout:4031 read 0xb843e59 thru 0xb84bf87 (0x812f bytes) 2024-04-24T00:36:11.227 INFO:teuthology.orchestra.run.smithi064.stdout:4032 writesame 0xb075da7 thru 0xb0792a8 data_size 0xa(0x3502 bytes) 2024-04-24T00:36:11.238 INFO:teuthology.orchestra.run.smithi064.stdout:4033 read 0x81f3093 thru 0x81f6c28 (0x3b96 bytes) 2024-04-24T00:36:11.238 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:11.238 INFO:teuthology.orchestra.run.smithi064.stdout:4037 write 0xa54fc7 thru 0xa58797 (0x37d1 bytes) 2024-04-24T00:36:11.263 INFO:teuthology.orchestra.run.smithi064.stdout:4038 write 0xe0d361f thru 0xe0df35d (0xbd3f bytes) 2024-04-24T00:36:11.271 INFO:teuthology.orchestra.run.smithi064.stdout:4039 read 0x179059e thru 0x1792f7e (0x29e1 bytes) 2024-04-24T00:36:11.271 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:11.264+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:11.272 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:11.272 INFO:teuthology.orchestra.run.smithi064.stdout:4041 writesame 0x5c2c4a thru 0x5c45c9 data_size 0x33(0x1980 bytes) 2024-04-24T00:36:11.295 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:11.295 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:11.295 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:11.295 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:11.295 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:11.300 INFO:teuthology.orchestra.run.smithi064.stdout:4042 trunc from 0xe0df35e to 0x1f402e1 2024-04-24T00:36:11.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:11.364 INFO:teuthology.orchestra.run.smithi064.stdout:4044 write 0x5dc0066 thru 0x5dcd738 (0xd6d3 bytes) 2024-04-24T00:36:11.372 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:11.367+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:11.376 INFO:teuthology.orchestra.run.smithi064.stdout:4045 read 0x198980f thru 0x19929c9 (0x91bb bytes) 2024-04-24T00:36:11.378 INFO:teuthology.orchestra.run.smithi064.stdout:4046 trunc from 0x5dcd739 to 0x642b500 2024-04-24T00:36:11.402 INFO:teuthology.orchestra.run.smithi064.stdout:4047 trunc from 0x642b500 to 0x2297346 2024-04-24T00:36:11.440 INFO:teuthology.orchestra.run.smithi064.stdout:4048 read 0x19cac9d thru 0x19d00b9 (0x541d bytes) 2024-04-24T00:36:11.440 INFO:teuthology.orchestra.run.smithi064.stdout:4049 read 0x1f8573a thru 0x1f86b52 (0x1419 bytes) 2024-04-24T00:36:11.473 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:11.464+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:11.476 INFO:teuthology.orchestra.run.smithi064.stdout:4050 write 0xba2e4ae thru 0xba355e7 (0x713a bytes) 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.266068508, 2024-04-24T00:36:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.26571759300001, 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.258829273, 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 67.000547647000005, 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.990271230999994, 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.528320428000001, 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:11.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:11.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.515912161999999, 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:11.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:11.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:11.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.517 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.515880408000001, 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.518 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:11.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:11.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:11.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81732 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2539520~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.512431552000002, 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042008566999999997, 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81732 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965398+0000", 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965400+0000", 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4150000000000002e-06 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965407+0000", 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0920000000000003e-06 2024-04-24T00:36:11.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965411+0000", 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7099999999999998e-06 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007262+0000", 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041850738999999998 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007328+0000", 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6375000000000001e-05 2024-04-24T00:36:11.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007406+0000", 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7844999999999998e-05 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81733 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2818048~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.512376015000001, 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042136290999999999, 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81733 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965453+0000", 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965454+0000", 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.37e-07 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965457+0000", 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3799999999999998e-06 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965466+0000", 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5280000000000003e-06 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965472+0000", 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4670000000000001e-06 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007435+0000", 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041963037000000002 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007502+0000", 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6727e-05 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007590+0000", 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7614999999999994e-05 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.512347912000003, 2024-04-24T00:36:11.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.502286407, 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:11.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.288343500000003, 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.278331555999998, 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:11.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:11.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:11.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.004952694, 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:11.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.988284244999999, 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:11.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:11.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.802386462000001, 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1186281050000009, 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0964111069999998, 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9351898270000003, 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:11.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:11.578 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:11.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:11.581 INFO:teuthology.orchestra.run.smithi064.stdout:4051 write 0xbd0ee7b thru 0xbd1e446 (0xf5cc bytes) 2024-04-24T00:36:11.581 INFO:teuthology.orchestra.run.smithi064.stdout:4052 writesame 0x18c4377 thru 0x18cb2e2 data_size 0x6(0x6f6c bytes) 2024-04-24T00:36:11.581 INFO:teuthology.orchestra.run.smithi064.stdout:4053 read 0x539f950 thru 0x53abc4a (0xc2fb bytes) 2024-04-24T00:36:11.581 INFO:teuthology.orchestra.run.smithi064.stdout:4055 punch from 0xb7e0349 to 0xb7e8fa3, (0x8c5a bytes) 2024-04-24T00:36:11.581 INFO:teuthology.orchestra.run.smithi064.stdout:4056 write 0x18729a1 thru 0x187dec5 (0xb525 bytes) 2024-04-24T00:36:11.581 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:11.581 INFO:teuthology.orchestra.run.smithi064.stdout:4058 clone 27 order 18 su 8192 sc 12 2024-04-24T00:36:11.582 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:11.575+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:11.681 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:11.672+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:11.714 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:11.715 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:11.715 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:11.715 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:11.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:11.782 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:11.779+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:11.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:11.914 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:11.911+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:11.984 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:11.980+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:11.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:12.041 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:12.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.822035352, 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:12.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:12.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:12.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:12.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.40024965800001, 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.047 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:12.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:12.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:12.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.3864267, 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:36:12.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:36:12.055 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.159285514999993, 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.056 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:36:12.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:12.061 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.258122092000001, 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:12.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:12.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:12.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.067 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.120350309999999, 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:12.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:12.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.082867991999997, 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.073 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.053539030000003, 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:36:12.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:12.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:12.084 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.856277126999998, 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:12.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.420522128000002, 2024-04-24T00:36:12.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:12.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:12.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.410664091999998, 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.390314484000001, 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.107 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.356269148999999, 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.353959313000001, 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.112 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:12.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:12.117 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.345390514000002, 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:12.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:12.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.406241421000001, 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:12.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:12.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:12.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:12.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.5168514250000005, 2024-04-24T00:36:12.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:12.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:12.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.4733756280000003, 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.136 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:12.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.499402326, 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:12.140 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.480260423, 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:12.146 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:12.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:12.150 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:12.151 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:12.152 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:12.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:12.153 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:12.147+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:12.159 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:12.159 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:12.178 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:12.178 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:12.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:12.195 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:12.215 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:12.215 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:12.235 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:12.235 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:12.252 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:12.244+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:12.253 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:12.253 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:12.272 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:12.272 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:12.322 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:12.322 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:12.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:12.382 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:12.379+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:12.455 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:12.452+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:12.456 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:12.456 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:12.471 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:12.556 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:12.552+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:12.641 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:12.641 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:12.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:12.656 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:12.651+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:12.757 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:12.752+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:12.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:36:12.786 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:12.786 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:12.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:12.858 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:12.852+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:12.958 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:12.952+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:13.009 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:13.009 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:13.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:13.082 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:13.079+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:13.162 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:13.152+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:13.215 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:13.216 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:13.239 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:13.262 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:13.256+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:13.363 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:13.356+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:13.455 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:13.455 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:13.463 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:13.456+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:13.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:13.564 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:13.556+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:13.605 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:13.605 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:13.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:13.665 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:13.660+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:13.730 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:13.731 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:13.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:13.766 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:13.763+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:36:13.867 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:13.863+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:13.895 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:13.895 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:13.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:13.956 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-04-24T00:36:13.968 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:13.963+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:14.035 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:14.035 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:14.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:14.093 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:14.088+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:14.169 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:14.160+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:14.206 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:14.206 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:14.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:14.263 INFO:teuthology.orchestra.run.smithi064.stdout:4059 punch from 0x73046b6 to 0x7311c5e, (0xd5a8 bytes) 2024-04-24T00:36:14.269 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:14.267+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:14.271 INFO:teuthology.orchestra.run.smithi064.stdout:4060 read 0x9b17f6a thru 0x9b25c51 (0xdce8 bytes) 2024-04-24T00:36:14.272 INFO:teuthology.orchestra.run.smithi064.stdout:4061 read 0xaec9a5e thru 0xaece53c (0x4adf bytes) 2024-04-24T00:36:14.274 INFO:teuthology.orchestra.run.smithi064.stdout:4062 read 0x1b07a0f thru 0x1b12fca (0xb5bc bytes) 2024-04-24T00:36:14.277 INFO:teuthology.orchestra.run.smithi064.stdout:4063 punch from 0x781c71c to 0x782a00a, (0xd8ee bytes) 2024-04-24T00:36:14.277 INFO:teuthology.orchestra.run.smithi064.stdout:4064 read 0x1d739a thru 0x1d792a (0x591 bytes) 2024-04-24T00:36:14.278 INFO:teuthology.orchestra.run.smithi064.stdout:4065 write 0xbb351cd thru 0xbb40147 (0xaf7b bytes) 2024-04-24T00:36:14.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:14.298 INFO:teuthology.orchestra.run.smithi064.stdout:4066 trunc from 0xbd1e447 to 0x197aac7 2024-04-24T00:36:14.371 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:14.367+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:14.403 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:14.403 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:14.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:14.431 INFO:teuthology.orchestra.run.smithi064.stdout:4067 trunc from 0x197aac7 to 0x7f6effc 2024-04-24T00:36:14.438 INFO:teuthology.orchestra.run.smithi064.stdout:4068 write 0xafcad1 thru 0xb03960 (0x6e90 bytes) 2024-04-24T00:36:14.442 INFO:teuthology.orchestra.run.smithi064.stdout:4069 punch from 0x2d0bfff to 0x2d1b1ee, (0xf1ef bytes) 2024-04-24T00:36:14.442 INFO:teuthology.orchestra.run.smithi064.stdout:4070 write 0x7b552db thru 0x7b564bd (0x11e3 bytes) 2024-04-24T00:36:14.447 INFO:teuthology.orchestra.run.smithi064.stdout:4071 read 0x1bad4f8 thru 0x1bb3c56 (0x675f bytes) 2024-04-24T00:36:14.447 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:14.454 INFO:teuthology.orchestra.run.smithi064.stdout:4073 write 0x80992b5 thru 0x809b9a2 (0x26ee bytes) 2024-04-24T00:36:14.460 INFO:teuthology.orchestra.run.smithi064.stdout:4074 read 0x5b6dfd2 thru 0x5b7a2b4 (0xc2e3 bytes) 2024-04-24T00:36:14.460 INFO:teuthology.orchestra.run.smithi064.stdout:4075 writesame 0x6b782b9 thru 0x6b83daf data_size 0x1(0xbaf7 bytes) 2024-04-24T00:36:14.462 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:14.462 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:14.473 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:14.471+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:14.481 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:14.482 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:14.497 INFO:teuthology.orchestra.run.smithi064.stdout:4076 write 0x2b48964 thru 0x2b55fed (0xd68a bytes) 2024-04-24T00:36:14.499 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:14.500 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:14.513 INFO:teuthology.orchestra.run.smithi064.stdout:4077 read 0x5f848a4 thru 0x5f8775b (0x2eb8 bytes) 2024-04-24T00:36:14.513 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:14.519 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:14.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:14.531 INFO:teuthology.orchestra.run.smithi064.stdout:4079 write 0xe636f81 thru 0xe63c4db (0x555b bytes) 2024-04-24T00:36:14.538 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:14.538 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:14.542 INFO:teuthology.orchestra.run.smithi064.stdout:4080 clone 28 order 23 su 4194304 sc 13 2024-04-24T00:36:14.557 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:14.557 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:14.571 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:14.572 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:14.572 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:14.572 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:14.572 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:14.573 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:14.567+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:14.586 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:14.587 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:14.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:14.607 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:14.608 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:14.673 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:14.671+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:14.727 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:14.727 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:14.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:14.727 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:14.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.727 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:14.727 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:14.727 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.289452983, 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:14.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:36:14.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.233941497, 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:36:14.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.133364849000003, 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:14.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:36:14.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:14.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.091799894999994, 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:36:14.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:36:14.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:36:14.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:36:14.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.528068333, 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:36:14.748 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:36:14.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:36:14.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.025094936000002, 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.754 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:36:14.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.786780702999998, 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:14.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:36:14.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751840280000003, 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:36:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:36:14.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:36:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751819406999999, 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:36:14.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:36:14.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:36:14.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:36:14.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:36:14.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:36:14.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:36:14.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.774 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.774 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.775 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:14.775 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751796028000001, 2024-04-24T00:36:14.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:36:14.775 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.775 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.776 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:36:14.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.776 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:14.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:14.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:36:14.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751774486000002, 2024-04-24T00:36:14.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:14.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:36:14.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:36:14.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:14.785 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751752965999998, 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:14.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:36:14.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:36:14.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751706808999998, 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.790 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:36:14.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751682690000003, 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751659285999999, 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:36:14.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:36:14.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751631513, 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751610100000001, 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.751589508999999, 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:14.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:36:14.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:36:14.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.535519266999998, 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:36:14.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:36:14.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.507548362999998, 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:36:14.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:14.824 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:14.825 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:14.825 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:14.825 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T00:36:14.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T00:36:14.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:14.827 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:14.824+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:14.926 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:14.920+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:15.007 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:15.008 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:15.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:15.027 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:15.020+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:15.127 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:15.120+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:15.183 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:36:15.183 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:15.183 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 151, 2024-04-24T00:36:15.183 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:36:15.183 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-04-24T00:33:24.857711+0000", 2024-04-24T00:36:15.183 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-04-24T00:36:14.558436+0000", 2024-04-24T00:36:15.183 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-04-24T00:35:14.574643+0000", 2024-04-24T00:36:15.183 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-04-24T00:35:24.656453+0000", 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5799936, 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 5, 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 3, 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-04-24T00:36:15.184 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:39.734814+0000", 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:15.185 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:15.186 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "17", 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:15.187 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:15.188 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:15.189 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 8, 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 8, 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.25, 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-04-24T00:36:15.190 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:43.786311+0000", 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:15.191 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:15.192 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "20", 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:36:15.193 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:15.194 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:15.195 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.5, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.5, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "pool_client.0", 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:51.288702+0000", 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:15.198 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "151", 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 58, 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 151, 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:15.199 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:15.200 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:36:15.201 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.75, 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.75, 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.059999942779541, 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.059999942779541, 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.202 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1c80b3f4-99ee-4725-90a8-920116161bf5", 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 13, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 74, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 95, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 53, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 75, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6817", 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6818", 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6819", 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6822", 2024-04-24T00:36:15.205 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6823", 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6820", 2024-04-24T00:36:15.206 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6821", 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6817/218259947", 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6819/218259947", 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6823/218259947", 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6821/218259947", 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:15.207 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "20e1331a-f8a2-48fd-b2b8-b7a92f21d78e", 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.264190673828125, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 122, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6824", 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6825", 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6826", 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6827", 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.210 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6830", 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6831", 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6828", 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6829", 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6825/637720411", 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6827/637720411", 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6831/637720411", 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6829/637720411", 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a51557a0-9634-4a20-82b0-b54cdd36b166", 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 129, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-04-24T00:36:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/820024667", 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6811/820024667", 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6815/820024667", 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6813/820024667", 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:15.217 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "aafffa79-f532-4441-b1d6-f6441c5eebf8", 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.79296875, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.3392791748046875, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 123, 2024-04-24T00:36:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.219 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/1974534323", 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6803/1974534323", 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1974534323", 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1974534323", 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:15.222 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "78f05556-598e-4be2-b682-e3f58863eb82", 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.9623870849609375, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 130, 2024-04-24T00:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6816", 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6818", 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.224 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6820", 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6821", 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6826", 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6828", 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6823", 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6824", 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6818/1642543604", 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6821/1642543604", 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6828/1642543604", 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6824/1642543604", 2024-04-24T00:36:15.227 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5", 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 142, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:15.228 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6808", 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6809", 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:15.229 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6810", 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6811", 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:15.230 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6814", 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6815", 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6812", 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6813", 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6809/172726194", 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6811/172726194", 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6815/172726194", 2024-04-24T00:36:15.232 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6813/172726194", 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "6dad12bf-94de-482a-a402-49e67d5e5dc7", 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:15.233 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 123, 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6800", 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.234 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6801", 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6802", 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6803", 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6806", 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6807", 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6804", 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6805", 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6801/2195746987", 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6803/2195746987", 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2195746987", 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6805/2195746987", 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.238 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e6734cfe-3c24-4ddd-8ef0-5ed450efe772", 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 62, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6817", 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6819", 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6822", 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6825", 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6830", 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6831", 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6827", 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6829", 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6819/954791674", 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6825/954791674", 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6831/954791674", 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6829/954791674", 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:15.243 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:15.244 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.509233+0000", 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 75 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.465696+0000", 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:15.245 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.405177+0000", 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.269202+0000", 2024-04-24T00:36:15.246 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.359180+0000", 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:15.247 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.351834+0000", 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.490127+0000", 2024-04-24T00:36:15.248 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.484842+0000", 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-04-24T00:36:15.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.3", 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: 3 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.4", 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: 7 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.250 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.16", 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: 2 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.1b", 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-04-24T00:36:15.251 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [ 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.252 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 55, 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 57, 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [ 2024-04-24T00:36:15.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 53, 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-04-24T00:36:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-04-24T00:36:15.255 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-04-24T00:36:15.255 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-04-24T00:36:15.255 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-04-24T00:36:15.255 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-04-24T00:36:15.255 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-04-24T00:36:15.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.255 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:15.257 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:15.257 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:15.257 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:15.257 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:15.257 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:15.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:15.259 INFO:tasks.thrashosds.thrasher:j is {'epoch': 151, 'fsid': '6a70d87d-4453-47be-af2a-5590f9851254', 'created': '2024-04-24T00:33:24.857711+0000', 'modified': '2024-04-24T00:36:14.558436+0000', 'last_up_change': '2024-04-24T00:35:14.574643+0000', 'last_in_change': '2024-04-24T00:35:24.656453+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T00:33:39.734814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T00:33:43.786311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-24T00:33:51.288702+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 58, 'snap_epoch': 151, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 0.75, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '1c80b3f4-99ee-4725-90a8-920116161bf5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 74, 'up_from': 95, 'up_thru': 53, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 218259947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 218259947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 218259947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 218259947}]}, 'public_addr': '172.21.15.64:6817/218259947', 'cluster_addr': '172.21.15.64:6819/218259947', 'heartbeat_back_addr': '172.21.15.64:6823/218259947', 'heartbeat_front_addr': '172.21.15.64:6821/218259947', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'up': 1, 'in': 1, 'weight': 0.264190673828125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 637720411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 637720411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 637720411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 637720411}]}, 'public_addr': '172.21.15.64:6825/637720411', 'cluster_addr': '172.21.15.64:6827/637720411', 'heartbeat_back_addr': '172.21.15.64:6831/637720411', 'heartbeat_front_addr': '172.21.15.64:6829/637720411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a51557a0-9634-4a20-82b0-b54cdd36b166', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 820024667}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 820024667}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 820024667}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 820024667}]}, 'public_addr': '172.21.15.64:6809/820024667', 'cluster_addr': '172.21.15.64:6811/820024667', 'heartbeat_back_addr': '172.21.15.64:6815/820024667', 'heartbeat_front_addr': '172.21.15.64:6813/820024667', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aafffa79-f532-4441-b1d6-f6441c5eebf8', 'up': 1, 'in': 1, 'weight': 0.79296875, 'primary_affinity': 0.3392791748046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1974534323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1974534323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1974534323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1974534323}]}, 'public_addr': '172.21.15.64:6801/1974534323', 'cluster_addr': '172.21.15.64:6803/1974534323', 'heartbeat_back_addr': '172.21.15.64:6807/1974534323', 'heartbeat_front_addr': '172.21.15.64:6805/1974534323', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '78f05556-598e-4be2-b682-e3f58863eb82', 'up': 1, 'in': 1, 'weight': 0.9623870849609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6816', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6818', 'nonce': 1642543604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6820', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6821', 'nonce': 1642543604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6826', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6828', 'nonce': 1642543604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6823', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6824', 'nonce': 1642543604}]}, 'public_addr': '172.21.15.167:6818/1642543604', 'cluster_addr': '172.21.15.167:6821/1642543604', 'heartbeat_back_addr': '172.21.15.167:6828/1642543604', 'heartbeat_front_addr': '172.21.15.167:6824/1642543604', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 142, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6808', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 172726194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6810', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 172726194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6814', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 172726194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6812', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 172726194}]}, 'public_addr': '172.21.15.167:6809/172726194', 'cluster_addr': '172.21.15.167:6811/172726194', 'heartbeat_back_addr': '172.21.15.167:6815/172726194', 'heartbeat_front_addr': '172.21.15.167:6813/172726194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6dad12bf-94de-482a-a402-49e67d5e5dc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6800', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2195746987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6802', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2195746987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6806', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2195746987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6804', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2195746987}]}, 'public_addr': '172.21.15.167:6801/2195746987', 'cluster_addr': '172.21.15.167:6803/2195746987', 'heartbeat_back_addr': '172.21.15.167:6807/2195746987', 'heartbeat_front_addr': '172.21.15.167:6805/2195746987', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e6734cfe-3c24-4ddd-8ef0-5ed450efe772', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6817', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6819', 'nonce': 954791674}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6822', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6825', 'nonce': 954791674}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6830', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6831', 'nonce': 954791674}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6827', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6829', 'nonce': 954791674}]}, 'public_addr': '172.21.15.167:6819/954791674', 'cluster_addr': '172.21.15.167:6825/954791674', 'heartbeat_back_addr': '172.21.15.167:6831/954791674', 'heartbeat_front_addr': '172.21.15.167:6829/954791674', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.509233+0000', 'dead_epoch': 75}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.465696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.405177+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.269202+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.359180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.351834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.490127+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.484842+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.3', 'osds': [7, 3]}, {'pgid': '2.4', 'osds': [5, 7]}, {'pgid': '3.16', 'osds': [3, 2]}, {'pgid': '3.1b', 'osds': [5, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 55, 'length': 1}, {'begin': 57, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 53, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T00:36:15.259 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.4 2024-04-24T00:36:15.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap 2.4 2024-04-24T00:36:15.260 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:15.248+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:15.357 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:15.352+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21437 3.5 3:a2bba8d3:::rbd_data.115eeb9272a5.000000000000001c:head [delete] snapc 10=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.853088067, 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071443490000000004, 2024-04-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21437 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583155+0000", 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583156+0000", 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3e-07 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583170+0000", 2024-04-24T00:36:15.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4876e-05 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583180+0000", 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.583187+0000", 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.19e-06 2024-04-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584786+0000", 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00159954 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.584922+0000", 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136052 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:26.590299+0000", 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053771169999999998 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.156393283, 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:33882 3.1f e71/59 3:faef3d65:::rbd_data.115ec4f9b3a7.0000000000000009:head v 71'131, mlcod=71'129)", 2024-04-24T00:36:15.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.350861323999993, 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080134579999999993, 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085382+0000", 2024-04-24T00:36:15.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085384+0000", 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7669999999999999e-06 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085551+0000", 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167083 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085588+0000", 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085607+0000", 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8479e-05 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085738+0000", 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013132000000000001 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.085789+0000", 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321999999999997e-05 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093318+0000", 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075291960000000002 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:47.093395+0000", 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7292000000000005e-05 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:77813 3.0 e102/21 3:05afdb1e:::rbd_data.115e805f80b3.0000000000000004:head v 102'237, mlcod=102'235)", 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.771420974999998, 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074761180000000003, 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664822+0000", 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664825+0000", 2024-04-24T00:36:15.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-04-24T00:36:15.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664911+0000", 2024-04-24T00:36:15.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5873999999999998e-05 2024-04-24T00:36:15.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664944+0000", 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.664959+0000", 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5979000000000001e-05 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665112+0000", 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152943 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.665157+0000", 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4725000000000003e-05 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672178+0000", 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070206390000000004 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.672298+0000", 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120308 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head, version: 45'133, data_included: [0~131072,196158~37038,262144~131072,524288~131072,786432~262144,1258308~52412,1441792~131072,3014656~131072,3267061~6628,6160384~34994], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head@45'133, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 6195378, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.514181892000003, 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011957290000000001, 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922063+0000", 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.306e-06 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923213+0000", 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011495310000000001 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923258+0000", 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4747999999999999e-05 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923284+0000", 2024-04-24T00:36:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5873000000000001e-05 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923436+0000", 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015226500000000001 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923452+0000", 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6022e-05 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.934018+0000", 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010566545 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head, version: 45'137, data_included: [0~131072,262144~131072,524288~131072,786432~199751,1179648~1552,1441792~3851,3014656~131072,5775043~64169,6160384~131072], data_size: 924683, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head@45'137, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8781824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.48535717, 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011581296, 2024-04-24T00:36:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950888+0000", 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951161+0000", 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027280499999999999 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951196+0000", 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123999999999998e-05 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951224+0000", 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8045e-05 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951356+0000", 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013187799999999999 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951376+0000", 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.003e-05 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.962467+0000", 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011091474 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.280022355, 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head, version: 80'319, data_included: [0~393216,524288~131072,1179648~524288], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head@80'319, size: 13631488, copy_subset: [(0, 13631488)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.267268262000002, 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089307889999999997, 2024-04-24T00:36:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168976+0000", 2024-04-24T00:36:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169620+0000", 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064358299999999998 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169650+0000", 2024-04-24T00:36:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0111e-05 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169671+0000", 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1231000000000001e-05 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169789+0000", 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118368 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169803+0000", 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3779e-05 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.177905+0000", 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081024800000000004 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.255121316, 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.238377157999999, 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.148228164999999, 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.741071485999999, 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.730684789000001, 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.7217415, 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:90104 3.18 3:193b8512:::rbd_data.115e56aa02fc.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write 1644318~44706] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.325608606999999, 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084175689999999997, 2024-04-24T00:36:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 90104 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110636+0000", 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110706+0000", 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9392000000000002e-05 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110734+0000", 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8538999999999999e-05 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110747+0000", 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860000000000001e-05 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110850+0000", 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102988 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111107+0000", 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025716600000000002 2024-04-24T00:36:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111346+0000", 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000238373 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.117684+0000", 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063379949999999999 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118724+0000", 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001040685 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118914+0000", 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018984699999999999 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.119052+0000", 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137701 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPushReply(3.12 128/127 [PushReplyOp(3:4e045fb2:::rbd_data.115e6f07caf6.0000000000000007:head)])", 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.058419237999999, 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089128709999999993, 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377827+0000", 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5870000000000001e-06 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377852+0000", 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5083e-05 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377890+0000", 2024-04-24T00:36:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7279e-05 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377914+0000", 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4496000000000001e-05 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.378154+0000", 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024028 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.386737+0000", 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085821460000000006 2024-04-24T00:36:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:92353 3.0 3:02b978df:::rbd_data.115ee57427d8.0000000000000008:head [truncate 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.6827789499999994, 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091800189999999993, 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 92353 2024-04-24T00:36:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753467+0000", 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7259999999999998e-06 2024-04-24T00:36:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753478+0000", 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1456e-05 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753503+0000", 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5015999999999999e-05 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753516+0000", 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.259e-05 2024-04-24T00:36:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753615+0000", 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9104999999999994e-05 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753624+0000", 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9309999999999997e-06 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755534+0000", 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019101700000000001 2024-04-24T00:36:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755620+0000", 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5736999999999996e-05 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755756+0000", 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013602800000000001 2024-04-24T00:36:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.756642+0000", 2024-04-24T00:36:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088663200000000005 2024-04-24T00:36:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762416+0000", 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057733849999999998 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762534+0000", 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118594 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762644+0000", 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010964900000000001 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:94309 3.0 3:06002a2d:::rbd_data.115eb3626b1c.000000000000002f:head [delete] snapc 37=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:10.447261+0000", 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9889824430000003, 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075631839999999997, 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 94309 2024-04-24T00:36:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447261+0000", 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447261+0000", 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447262+0000", 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.553e-06 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447268+0000", 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4559999999999999e-06 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447282+0000", 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4763000000000001e-05 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447289+0000", 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1720000000000001e-06 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447337+0000", 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7184000000000001e-05 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447396+0000", 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9614999999999998e-05 2024-04-24T00:36:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447458+0000", 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1284000000000005e-05 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.447968+0000", 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000510562 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.454763+0000", 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067951620000000004 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.454793+0000", 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9448999999999999e-05 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.454824+0000", 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0984e-05 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:94358 3.0 3:06002a2d:::rbd_data.115eb3626b1c.000000000000002f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2512793~56897] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6971791380000001, 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010252809999999999, 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 94358 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739067+0000", 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9299999999999999e-06 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739156+0000", 2024-04-24T00:36:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9148999999999996e-05 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739189+0000", 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2772999999999999e-05 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739205+0000", 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6722999999999999e-05 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739285+0000", 2024-04-24T00:36:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9729999999999997e-05 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739348+0000", 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3309e-05 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739445+0000", 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6378000000000005e-05 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.742520+0000", 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030747819999999999 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749133+0000", 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066130659999999999 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749238+0000", 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010506299999999999 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749317+0000", 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8906999999999995e-05 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:94675 3.0 3:061b3b47:::rbd_data.115eb3626b1c.0000000000000003:head [write 2173345~46373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:11.531466+0000", 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.904777079, 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075444240000000001, 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 94675 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531466+0000", 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531466+0000", 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531469+0000", 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.807e-06 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531515+0000", 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6609e-05 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531553+0000", 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8099999999999998e-05 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531577+0000", 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3646999999999999e-05 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531676+0000", 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9308000000000005e-05 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531774+0000", 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7404000000000005e-05 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.531914+0000", 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014038399999999999 2024-04-24T00:36:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.535657+0000", 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037425980000000002 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.538913+0000", 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032565319999999999 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.538966+0000", 2024-04-24T00:36:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3140999999999998e-05 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:11.539010+0000", 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3893999999999997e-05 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.537 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:15.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:15.539 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:15.535+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:15.640 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:15.635+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.043246200999999998, 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043294257000000003, 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.030621241, 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030728086000000002, 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105174 3.1e e152/44)", 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.718983+0000", 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.010251598000000001, 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010400428999999999, 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.718983+0000", 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.718983+0000", 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.718983+0000", 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.723467+0000", 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004484025 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.723477+0000", 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9380000000000004e-06 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.723485+0000", 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2090000000000005e-06 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105283 3.16 3.e153896 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.726946+0000", 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0022883280000000001, 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024687950000000002, 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105283 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.726946+0000", 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.726946+0000", 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.726947+0000", 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3319999999999999e-06 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.726953+0000", 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4450000000000004e-06 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.726971+0000", 2024-04-24T00:36:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8332999999999999e-05 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.726978+0000", 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0299999999999996e-06 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105318 3.16 3.f433c6f6 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.727013+0000", 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0022213929999999999, 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024392440000000001, 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105318 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.727013+0000", 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.727013+0000", 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.727013+0000", 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.727016+0000", 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4490000000000002e-06 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.727022+0000", 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2190000000000001e-06 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.727027+0000", 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2510000000000002e-06 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 5 2024-04-24T00:36:15.752 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:15.754 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:15.751+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:15.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:15.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:36:15.857 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:15.852+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:15.871 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:15.871 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:15.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:15.955 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:15.951+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:16.037 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.822340434, 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.821989519, 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.815101199, 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 71.556819572999999, 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 71.546543157000002, 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.084592354000002, 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.072184088, 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.072152334000002, 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.068619837999996, 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.058558333000001, 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:16.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:16.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:16.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.844615425999997, 2024-04-24T00:36:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:16.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:16.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.834603481999999, 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:16.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.561224619999997, 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:16.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:16.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.544556170999996, 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:16.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.358658387999998, 2024-04-24T00:36:16.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:16.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:16.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.674900031, 2024-04-24T00:36:16.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:16.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.652683033000001, 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:16.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.491461752999999, 2024-04-24T00:36:16.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:16.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:16.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.34811319499999999, 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:36:16.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:16.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.33548823500000002, 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:16.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:16.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:16.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:16.220 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:16.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:16.222 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:16.219+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:16.322 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:16.319+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:16.351 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:16.352 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:16.352 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:16.352 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:16.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:16.454 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:16.451+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:16.524 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:16.520+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:16.536 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:16.537 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:16.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:16.567 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:36:16.577 INFO:teuthology.orchestra.run.smithi064.stderr:clear 2.4 pg_upmap mapping 2024-04-24T00:36:16.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:16.641 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:16.639+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:16.718 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.49976024, 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.719 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:16.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:16.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:16.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:16.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:16.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:16.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:16.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:16.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:16.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:16.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:16.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:16.727 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.727 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.727 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.727 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:16.728 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:16.728 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.07797454599999, 2024-04-24T00:36:16.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:16.728 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.728 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.729 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:16.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.729 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:16.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.730 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:16.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:16.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:16.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:16.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:16.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:21646 3.1d e52/21 ondisk, result = 0)", 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.064151588, 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067632399999999995, 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649934+0000", 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649935+0000", 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5600000000000001e-06 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649943+0000", 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5229999999999998e-06 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649969+0000", 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6256000000000001e-05 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.649982+0000", 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.332e-05 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717536+0000", 2024-04-24T00:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067554264000000003 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717547+0000", 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0762e-05 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717566+0000", 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8715e-05 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22567 3.14 3:285794c6:::rbd_data.115e6a380959.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 98.837010402999994, 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066610388000000006, 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22567 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877075+0000", 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877076+0000", 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7239999999999999e-06 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877086+0000", 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3959999999999995e-06 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877111+0000", 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4896999999999999e-05 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877126+0000", 2024-04-24T00:36:16.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4857e-05 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877214+0000", 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8242e-05 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.877337+0000", 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012321200000000001 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 7", 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.880303+0000", 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029661510000000002 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943550+0000", 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063246347999999994 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943601+0000", 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1819e-05 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943644+0000", 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2259000000000003e-05 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.943685+0000", 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1483000000000002e-05 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.935846980000001, 2024-04-24T00:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:16.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:16.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:16.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:16.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:16.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:16.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:16.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:16.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:16.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:16.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.798075197999999, 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.753 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:16.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:16.758 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.760592879999997, 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:16.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:16.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:16.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:16.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.763 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.731263918000003, 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:36:16.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.769 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.534002014999999, 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:16.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:16.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:16.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:16.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 39.098247016000002, 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:16.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 39.088388979999998, 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:16.781 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:16.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:16.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:16.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 39.068039372000001, 2024-04-24T00:36:16.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:16.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:16.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.033994036999999, 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:16.794 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:16.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:16.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.031684201000001, 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.799 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:16.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:16.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.023115401999998, 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.805 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:16.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:16.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.083966309000001, 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:16.811 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:16.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:16.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:16.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 11.194576313000001, 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:16.819 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:16.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 11.151100516, 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:16.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:16.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:16.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.1771272140000004, 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.829 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:16.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.157985311, 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:16.835 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:16.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:16.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:16.838 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.839 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.839 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.839 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:16.839 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:16.839 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:16.840 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-04-24T00:36:16.840 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:16.841 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:16.832+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:16.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:16.941 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:16.939+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:17.014 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:17.014 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:17.014 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:17.015 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:17.028 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:17.033 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:17.033 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:17.042 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:17.036+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:17.061 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:17.062 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:17.092 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:17.092 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:17.098 INFO:teuthology.orchestra.run.smithi064.stdout:4081 writesame 0xb9e1a78 thru 0xb9f0a7c data_size 0x1(0xf005 bytes) 2024-04-24T00:36:17.114 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:17.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:17.143 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:17.139+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:17.143 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:17.144 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:17.166 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:17.167 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:17.190 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:17.191 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:17.197 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:17.197 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:17.212 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:17.254 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:17.251+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:17.256 INFO:teuthology.orchestra.run.smithi064.stdout:4082 read 0x1fec3aa thru 0x1ff604f (0x9ca6 bytes) 2024-04-24T00:36:17.256 INFO:teuthology.orchestra.run.smithi064.stdout:4084 read 0xd53b92a thru 0xd5413a1 (0x5a78 bytes) 2024-04-24T00:36:17.256 INFO:teuthology.orchestra.run.smithi064.stdout:4085 write 0x7901983 thru 0x79079e5 (0x6063 bytes) 2024-04-24T00:36:17.261 INFO:teuthology.orchestra.run.smithi064.stdout:4087 punch from 0x98844f0 to 0x9886edc, (0x29ec bytes) 2024-04-24T00:36:17.261 INFO:teuthology.orchestra.run.smithi064.stdout:4088 punch from 0x7c7ffe2 to 0x7c88ba4, (0x8bc2 bytes) 2024-04-24T00:36:17.261 INFO:teuthology.orchestra.run.smithi064.stdout:4089 punch from 0xfe161d to 0xfe3497, (0x1e7a bytes) 2024-04-24T00:36:17.261 INFO:teuthology.orchestra.run.smithi064.stdout:4090 read 0x1bea842 thru 0x1bf597e (0xb13d bytes) 2024-04-24T00:36:17.262 INFO:teuthology.orchestra.run.smithi064.stdout:4091 read 0x33787fa thru 0x3383fbb (0xb7c2 bytes) 2024-04-24T00:36:17.262 INFO:teuthology.orchestra.run.smithi064.stdout:4092 writesame 0x7813976 thru 0x781cb31 data_size 0x6(0x91bc bytes) 2024-04-24T00:36:17.290 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.290 INFO:teuthology.orchestra.run.smithi064.stdout:4094 write 0x2ee0840 thru 0x2eee454 (0xdc15 bytes) 2024-04-24T00:36:17.298 INFO:teuthology.orchestra.run.smithi064.stdout:4099 read 0x9a1b01c thru 0x9a25607 (0xa5ec bytes) 2024-04-24T00:36:17.298 INFO:teuthology.orchestra.run.smithi064.stdout:4101 writesame 0x7a8c0cf thru 0x7a964d2 data_size 0x6(0xa404 bytes) 2024-04-24T00:36:17.311 INFO:teuthology.orchestra.run.smithi064.stdout:4102 read 0x2f61b27 thru 0x2f7159e (0xfa78 bytes) 2024-04-24T00:36:17.312 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.312 INFO:teuthology.orchestra.run.smithi064.stdout:4107 write 0x9c2af0e thru 0x9c32508 (0x75fb bytes) 2024-04-24T00:36:17.319 INFO:teuthology.orchestra.run.smithi064.stdout:4108 punch from 0xbb288ad to 0xbb2918b, (0x8de bytes) 2024-04-24T00:36:17.324 INFO:teuthology.orchestra.run.smithi064.stdout:4109 trunc from 0xe63c4dc to 0xc561f28 2024-04-24T00:36:17.344 INFO:teuthology.orchestra.run.smithi064.stdout:4110 read 0x5f3fb0c thru 0x5f4272c (0x2c21 bytes) 2024-04-24T00:36:17.345 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:17.339+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:17.345 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.345 INFO:teuthology.orchestra.run.smithi064.stdout:4113 read 0xa0c6104 thru 0xa0ce944 (0x8841 bytes) 2024-04-24T00:36:17.345 INFO:teuthology.orchestra.run.smithi064.stdout:4115 writesame 0x7163763 thru 0x7173664 data_size 0x7(0xff02 bytes) 2024-04-24T00:36:17.381 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.383 INFO:teuthology.orchestra.run.smithi064.stdout:4118 trunc from 0xc561f28 to 0xd3c5fd3 2024-04-24T00:36:17.385 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:17.385 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:17.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:17.415 INFO:teuthology.orchestra.run.smithi064.stdout:4119 write 0xe085df1 thru 0xe085e5a (0x6a bytes) 2024-04-24T00:36:17.438 INFO:teuthology.orchestra.run.smithi064.stdout:4121 trunc from 0xe085e5b to 0x85b7dc8 2024-04-24T00:36:17.446 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:17.444+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:17.471 INFO:teuthology.orchestra.run.smithi064.stdout:4123 write 0x806ae08 thru 0x806aec5 (0xbe bytes) 2024-04-24T00:36:17.497 INFO:teuthology.orchestra.run.smithi064.stdout:4125 trunc from 0x85b7dc8 to 0x1b1246 2024-04-24T00:36:17.547 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:17.540+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:17.556 INFO:teuthology.orchestra.run.smithi064.stdout:4126 punch from 0x663b5 to 0x736f6, (0xd341 bytes) 2024-04-24T00:36:17.556 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.556 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.556 INFO:teuthology.orchestra.run.smithi064.stdout:4131 punch from 0x143e00 to 0x151f89, (0xe189 bytes) 2024-04-24T00:36:17.580 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:17.580 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:17.588 INFO:teuthology.orchestra.run.smithi064.stdout:4132 write 0x6024c25 thru 0x602ce9e (0x827a bytes) 2024-04-24T00:36:17.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:17.597 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.597 INFO:teuthology.orchestra.run.smithi064.stdout:4135 read 0xa2c113 thru 0xa3670e (0xa5fc bytes) 2024-04-24T00:36:17.597 INFO:teuthology.orchestra.run.smithi064.stdout:4136 punch from 0x58b6504 to 0x58ba7c0, (0x42bc bytes) 2024-04-24T00:36:17.600 INFO:teuthology.orchestra.run.smithi064.stdout:4137 trunc from 0x602ce9f to 0x4ab4c04 2024-04-24T00:36:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:4138 read 0x43d560e thru 0x43e19a1 (0xc394 bytes) 2024-04-24T00:36:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:4139 punch from 0x1fb52e5 to 0x1fc3827, (0xe542 bytes) 2024-04-24T00:36:17.642 INFO:teuthology.orchestra.run.smithi064.stdout:4141 writesame 0x7ff6b9e thru 0x8001b2e data_size 0x59(0xaf91 bytes) 2024-04-24T00:36:17.642 INFO:teuthology.orchestra.run.smithi064.stdout:4142 read 0x54b23d7 thru 0x54b7253 (0x4e7d bytes) 2024-04-24T00:36:17.642 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.642 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.642 INFO:teuthology.orchestra.run.smithi064.stdout:4146 punch from 0x1b3c0cd to 0x1b41f4f, (0x5e82 bytes) 2024-04-24T00:36:17.648 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:17.640+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:17.649 INFO:teuthology.orchestra.run.smithi064.stdout:4147 trunc from 0x8001b2f to 0x576dd97 2024-04-24T00:36:17.685 INFO:teuthology.orchestra.run.smithi064.stdout:4151 writesame 0x7a32477 thru 0x7a3379b data_size 0xd(0x1325 bytes) 2024-04-24T00:36:17.709 INFO:teuthology.orchestra.run.smithi064.stdout:4152 writesame 0xaed3dca thru 0xaed9afe data_size 0x1(0x5d35 bytes) 2024-04-24T00:36:17.723 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:17.723 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:17.732 INFO:teuthology.orchestra.run.smithi064.stdout:4153 write 0xe3a6406 thru 0xe3b3322 (0xcf1d bytes) 2024-04-24T00:36:17.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:17.740 INFO:teuthology.orchestra.run.smithi064.stdout:4155 write 0x4a9a90c thru 0x4aa69dc (0xc0d1 bytes) 2024-04-24T00:36:17.753 INFO:teuthology.orchestra.run.smithi064.stdout:4156 punch from 0x5f4d214 to 0x5f52310, (0x50fc bytes) 2024-04-24T00:36:17.754 INFO:teuthology.orchestra.run.smithi064.stdout:4158 write 0xd178c8 thru 0xd21775 (0x9eae bytes) 2024-04-24T00:36:17.762 INFO:teuthology.orchestra.run.smithi064.stdout:4160 punch from 0x62ec2ee to 0x62ecd66, (0xa78 bytes) 2024-04-24T00:36:17.762 INFO:teuthology.orchestra.run.smithi064.stdout:4161 punch from 0x4628eb8 to 0x463188b, (0x89d3 bytes) 2024-04-24T00:36:17.767 INFO:teuthology.orchestra.run.smithi064.stdout:4163 trunc from 0xe3b3323 to 0xc4b1920 2024-04-24T00:36:17.785 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:17.783+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:17.791 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.798 INFO:teuthology.orchestra.run.smithi064.stdout:4165 trunc from 0xc4b1920 to 0x9b54c50 2024-04-24T00:36:17.812 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.812 INFO:teuthology.orchestra.run.smithi064.stdout:4167 read 0x7f1a630 thru 0x7f2a0d0 (0xfaa1 bytes) 2024-04-24T00:36:17.812 INFO:teuthology.orchestra.run.smithi064.stdout:4168 write 0x122b322 thru 0x1232e78 (0x7b57 bytes) 2024-04-24T00:36:17.819 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.819 INFO:teuthology.orchestra.run.smithi064.stdout:4170 writesame 0x2bc3227 thru 0x2bce185 data_size 0x7b(0xaf5f bytes) 2024-04-24T00:36:17.825 INFO:teuthology.orchestra.run.smithi064.stdout:4171 punch from 0x42aa513 to 0x42b3ed9, (0x99c6 bytes) 2024-04-24T00:36:17.825 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.825 INFO:teuthology.orchestra.run.smithi064.stdout:4175 write 0x5944c8 thru 0x59cadb (0x8614 bytes) 2024-04-24T00:36:17.835 INFO:teuthology.orchestra.run.smithi064.stdout:4176 write 0x94b227e thru 0x94bdc26 (0xb9a9 bytes) 2024-04-24T00:36:17.846 INFO:teuthology.orchestra.run.smithi064.stdout:4178 write 0x8ac0586 thru 0x8acc2ef (0xbd6a bytes) 2024-04-24T00:36:17.850 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:17.844+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:17.859 INFO:teuthology.orchestra.run.smithi064.stdout:4179 read 0x426776f thru 0x426db9f (0x6431 bytes) 2024-04-24T00:36:17.860 INFO:teuthology.orchestra.run.smithi064.stdout:4180 write 0x3dac6b thru 0x3dd76c (0x2b02 bytes) 2024-04-24T00:36:17.864 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:17.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:17.885 INFO:teuthology.orchestra.run.smithi064.stdout:4183 writesame 0xb1e9834 thru 0xb1ee9f5 data_size 0x23(0x51c2 bytes) 2024-04-24T00:36:17.885 INFO:teuthology.orchestra.run.smithi064.stdout:4186 write 0x67a11a0 thru 0x67a7409 (0x626a bytes) 2024-04-24T00:36:17.897 INFO:teuthology.orchestra.run.smithi064.stdout:4187 punch from 0x10ab855 to 0x10b8b28, (0xd2d3 bytes) 2024-04-24T00:36:17.897 INFO:teuthology.orchestra.run.smithi064.stdout:4188 write 0x82f150a thru 0x82fc86b (0xb362 bytes) 2024-04-24T00:36:17.909 INFO:teuthology.orchestra.run.smithi064.stdout:4189 punch from 0x739b523 to 0x73a9cf2, (0xe7cf bytes) 2024-04-24T00:36:17.909 INFO:teuthology.orchestra.run.smithi064.stdout:4190 read 0x6b71ac5 thru 0x6b7961b (0x7b57 bytes) 2024-04-24T00:36:17.909 INFO:teuthology.orchestra.run.smithi064.stdout:4191 read 0x7aba1c2 thru 0x7ac21b8 (0x7ff7 bytes) 2024-04-24T00:36:17.909 INFO:teuthology.orchestra.run.smithi064.stdout:4192 write 0x53373fb thru 0x5340393 (0x8f99 bytes) 2024-04-24T00:36:17.924 INFO:teuthology.orchestra.run.smithi064.stdout:4193 trunc from 0xb1ee9f6 to 0xc8c1da6 2024-04-24T00:36:17.937 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:17.951 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:17.948+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:17.961 INFO:teuthology.orchestra.run.smithi064.stdout:4195 trunc from 0xc8c1da6 to 0x3e6bd5d 2024-04-24T00:36:17.993 INFO:teuthology.orchestra.run.smithi064.stdout:4196 trunc from 0x3e6bd5d to 0x286311b 2024-04-24T00:36:18.036 INFO:teuthology.orchestra.run.smithi064.stdout:4197 writesame 0x408613b thru 0x408cc96 data_size 0x2(0x6b5c bytes) 2024-04-24T00:36:18.050 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:18.050 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:18.052 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:18.047+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:18.056 INFO:teuthology.orchestra.run.smithi064.stdout:4198 writesame 0x582ebe5 thru 0x583db5e data_size 0x97(0xef7a bytes) 2024-04-24T00:36:18.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:18.093 INFO:teuthology.orchestra.run.smithi064.stdout:4199 write 0xdcf7a06 thru 0xdd01619 (0x9c14 bytes) 2024-04-24T00:36:18.100 INFO:teuthology.orchestra.run.smithi064.stdout:4201 write 0x91edd57 thru 0x91ee70e (0x9b8 bytes) 2024-04-24T00:36:18.107 INFO:teuthology.orchestra.run.smithi064.stdout:4203 punch from 0x4cdc2b5 to 0x4cdd466, (0x11b1 bytes) 2024-04-24T00:36:18.107 INFO:teuthology.orchestra.run.smithi064.stdout:4204 read 0xc73ded8 thru 0xc749480 (0xb5a9 bytes) 2024-04-24T00:36:18.108 INFO:teuthology.orchestra.run.smithi064.stdout:4205 punch from 0xbd6ede4 to 0xbd70239, (0x1455 bytes) 2024-04-24T00:36:18.108 INFO:teuthology.orchestra.run.smithi064.stdout:4207 writesame 0x4c5fc8a thru 0x4c6c633 data_size 0x2(0xc9aa bytes) 2024-04-24T00:36:18.158 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:18.155+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:18.218 INFO:teuthology.orchestra.run.smithi064.stdout:4208 writesame 0x84d58c7 thru 0x84e2170 data_size 0x37(0xc8aa bytes) 2024-04-24T00:36:18.226 INFO:teuthology.orchestra.run.smithi064.stdout:4209 write 0xb216a1e thru 0xb224f01 (0xe4e4 bytes) 2024-04-24T00:36:18.232 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:18.232 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:18.238 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:18.253 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:18.248+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:18.272 INFO:teuthology.orchestra.run.smithi064.stdout:4211 trunc from 0xdd0161a to 0x7f8660 2024-04-24T00:36:18.314 INFO:teuthology.orchestra.run.smithi064.stdout:4213 read 0x7a1abd thru 0x7a4f61 (0x34a5 bytes) 2024-04-24T00:36:18.315 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.315 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.336 INFO:teuthology.orchestra.run.smithi064.stdout:4219 write 0x3c0e377 thru 0x3c11908 (0x3592 bytes) 2024-04-24T00:36:18.344 INFO:teuthology.orchestra.run.smithi064.stdout:4220 read 0x253eaff thru 0x2546fda (0x84dc bytes) 2024-04-24T00:36:18.344 INFO:teuthology.orchestra.run.smithi064.stdout:4221 punch from 0x1b57475 to 0x1b5f561, (0x80ec bytes) 2024-04-24T00:36:18.344 INFO:teuthology.orchestra.run.smithi064.stdout:4222 punch from 0x195a142 to 0x195eff8, (0x4eb6 bytes) 2024-04-24T00:36:18.344 INFO:teuthology.orchestra.run.smithi064.stdout:4223 trunc from 0x3c11909 to 0x38f3258 2024-04-24T00:36:18.354 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:18.352+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:18.360 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.360 INFO:teuthology.orchestra.run.smithi064.stdout:4225 write 0x22f6c5e thru 0x22ff848 (0x8beb bytes) 2024-04-24T00:36:18.371 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.371 INFO:teuthology.orchestra.run.smithi064.stdout:4231 read 0xb04703 thru 0xb0d928 (0x9226 bytes) 2024-04-24T00:36:18.372 INFO:teuthology.orchestra.run.smithi064.stdout:4232 read 0x9fc207 thru 0x9fee6d (0x2c67 bytes) 2024-04-24T00:36:18.373 INFO:teuthology.orchestra.run.smithi064.stdout:4234 punch from 0x1ae2fab to 0x1ae8908, (0x595d bytes) 2024-04-24T00:36:18.373 INFO:teuthology.orchestra.run.smithi064.stdout:4236 write 0x110a6ae thru 0x110fb57 (0x54aa bytes) 2024-04-24T00:36:18.409 INFO:teuthology.orchestra.run.smithi064.stdout:4237 write 0x8299386 thru 0x82a604d (0xccc8 bytes) 2024-04-24T00:36:18.416 INFO:teuthology.orchestra.run.smithi064.stdout:4238 read 0xa5c13c thru 0xa63467 (0x732c bytes) 2024-04-24T00:36:18.418 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:18.418 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:18.432 INFO:teuthology.orchestra.run.smithi064.stdout:4239 trunc from 0x82a604e to 0xd34b73f 2024-04-24T00:36:18.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:18.446 INFO:teuthology.orchestra.run.smithi064.stdout:4240 punch from 0xaa83ea6 to 0xaa906ee, (0xc848 bytes) 2024-04-24T00:36:18.446 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.446 INFO:teuthology.orchestra.run.smithi064.stdout:4242 writesame 0x4b7acf3 thru 0x4b7d6f1 data_size 0x1(0x29ff bytes) 2024-04-24T00:36:18.454 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:18.451+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:18.490 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.490 INFO:teuthology.orchestra.run.smithi064.stdout:4244 read 0x5e9f237 thru 0x5eae54e (0xf318 bytes) 2024-04-24T00:36:18.505 INFO:teuthology.orchestra.run.smithi064.stdout:4245 trunc from 0xd34b73f to 0x5675716 2024-04-24T00:36:18.535 INFO:teuthology.orchestra.run.smithi064.stdout:4246 trunc from 0x5675716 to 0x962b3bf 2024-04-24T00:36:18.545 INFO:teuthology.orchestra.run.smithi064.stdout:4248 read 0xdd9380 thru 0xddfc5b (0x68dc bytes) 2024-04-24T00:36:18.545 INFO:teuthology.orchestra.run.smithi064.stdout:4251 writesame 0x93a7715 thru 0x93aec7c data_size 0x44(0x7568 bytes) 2024-04-24T00:36:18.558 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:18.555+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:18.573 INFO:teuthology.orchestra.run.smithi064.stdout:4252 write 0xe513a29 thru 0xe523a0f (0xffe7 bytes) 2024-04-24T00:36:18.575 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:18.575 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:18.580 INFO:teuthology.orchestra.run.smithi064.stdout:4253 read 0x7d1bde7 thru 0x7d1e09e (0x22b8 bytes) 2024-04-24T00:36:18.580 INFO:teuthology.orchestra.run.smithi064.stdout:4254 writesame 0x26ee578 thru 0x26fd2eb data_size 0x34(0xed74 bytes) 2024-04-24T00:36:18.589 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:18.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:36:18.611 INFO:teuthology.orchestra.run.smithi064.stdout:4257 trunc from 0xe523a10 to 0xeab87e 2024-04-24T00:36:18.659 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:18.656+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:18.676 INFO:teuthology.orchestra.run.smithi064.stdout:4258 write 0xca8784e thru 0xca95538 (0xdceb bytes) 2024-04-24T00:36:18.682 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.683 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.683 INFO:teuthology.orchestra.run.smithi064.stdout:4261 read 0xc1a57e6 thru 0xc1ace7c (0x7697 bytes) 2024-04-24T00:36:18.683 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.683 INFO:teuthology.orchestra.run.smithi064.stdout:4263 writesame 0x9984809 thru 0x9984902 data_size 0x1(0xfa bytes) 2024-04-24T00:36:18.690 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.690 INFO:teuthology.orchestra.run.smithi064.stdout:4265 punch from 0x26d4d09 to 0x26e215d, (0xd454 bytes) 2024-04-24T00:36:18.690 INFO:teuthology.orchestra.run.smithi064.stdout:4266 punch from 0x2e8939d to 0x2e90827, (0x748a bytes) 2024-04-24T00:36:18.690 INFO:teuthology.orchestra.run.smithi064.stdout:4267 read 0x62eff34 thru 0x62f55da (0x56a7 bytes) 2024-04-24T00:36:18.694 INFO:teuthology.orchestra.run.smithi064.stdout:4268 trunc from 0xca95539 to 0x9f0b7e0 2024-04-24T00:36:18.705 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.718 INFO:teuthology.orchestra.run.smithi064.stdout:4273 writesame 0xc8fce31 thru 0xc8ff74d data_size 0x5(0x291d bytes) 2024-04-24T00:36:18.721 INFO:teuthology.orchestra.run.smithi064.stdout:4274 writesame 0x52704a0 thru 0x5273ff1 data_size 0x3(0x3b52 bytes) 2024-04-24T00:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.742 INFO:teuthology.orchestra.run.smithi064.stdout:4278 write 0x1f78349 thru 0x1f78cb0 (0x968 bytes) 2024-04-24T00:36:18.745 INFO:teuthology.orchestra.run.smithi064.stdout:4279 write 0xb71f299 thru 0xb7280c9 (0x8e31 bytes) 2024-04-24T00:36:18.755 INFO:teuthology.orchestra.run.smithi064.stdout:4280 writesame 0x224a6ff thru 0x225105b data_size 0x51(0x695d bytes) 2024-04-24T00:36:18.757 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:18.757 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:18.762 INFO:teuthology.orchestra.run.smithi064.stdout:4281 read 0x2baab1f thru 0x2bb892c (0xde0e bytes) 2024-04-24T00:36:18.762 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.762 INFO:teuthology.orchestra.run.smithi064.stdout:4284 write 0xa81b1fe thru 0xa8263bc (0xb1bf bytes) 2024-04-24T00:36:18.762 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:18.760+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:18.772 INFO:teuthology.orchestra.run.smithi064.stdout:4285 write 0x67e003d thru 0x67eefc1 (0xef85 bytes) 2024-04-24T00:36:18.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:18.785 INFO:teuthology.orchestra.run.smithi064.stdout:4286 punch from 0x8dbe1d7 to 0x8dbf19a, (0xfc3 bytes) 2024-04-24T00:36:18.785 INFO:teuthology.orchestra.run.smithi064.stdout:4287 write 0xc536591 thru 0xc543bee (0xd65e bytes) 2024-04-24T00:36:18.797 INFO:teuthology.orchestra.run.smithi064.stdout:4288 read 0x4bd184b thru 0x4bdfbfb (0xe3b1 bytes) 2024-04-24T00:36:18.797 INFO:teuthology.orchestra.run.smithi064.stdout:4289 writesame 0x56e5d2c thru 0x56ec314 data_size 0x1(0x65e9 bytes) 2024-04-24T00:36:18.856 INFO:teuthology.orchestra.run.smithi064.stdout:4290 writesame 0x53051f6 thru 0x5305d1e data_size 0x1(0xb29 bytes) 2024-04-24T00:36:18.863 INFO:teuthology.orchestra.run.smithi064.stdout:4295 punch from 0x2f6530e to 0x2f672a2, (0x1f94 bytes) 2024-04-24T00:36:18.863 INFO:teuthology.orchestra.run.smithi064.stdout:4296 write 0xac125a4 thru 0xac18191 (0x5bee bytes) 2024-04-24T00:36:18.863 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:18.860+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:18.881 INFO:teuthology.orchestra.run.smithi064.stdout:4297 writesame 0xdda3c37 thru 0xdda8b25 data_size 0xb(0x4eef bytes) 2024-04-24T00:36:18.881 INFO:teuthology.orchestra.run.smithi064.stdout:4298 writesame 0x798ffbf thru 0x7990678 data_size 0x15(0x6ba bytes) 2024-04-24T00:36:18.900 INFO:teuthology.orchestra.run.smithi064.stdout:4299 trunc from 0xdda8b26 to 0x461be3d 2024-04-24T00:36:18.911 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:18.911 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:18.923 INFO:teuthology.orchestra.run.smithi064.stdout:4300 read 0x1f1799b thru 0x1f24f27 (0xd58d bytes) 2024-04-24T00:36:18.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:18.933 INFO:teuthology.orchestra.run.smithi064.stdout:4301 write 0x50bdd5b thru 0x50bf539 (0x17df bytes) 2024-04-24T00:36:18.937 INFO:teuthology.orchestra.run.smithi064.stdout:4302 read 0x19d2c19 thru 0x19d49ff (0x1de7 bytes) 2024-04-24T00:36:18.937 INFO:teuthology.orchestra.run.smithi064.stdout:4303 read 0x4c00790 thru 0x4c0c8b0 (0xc121 bytes) 2024-04-24T00:36:18.956 INFO:teuthology.orchestra.run.smithi064.stdout:4304 write 0xc276218 thru 0xc27679d (0x586 bytes) 2024-04-24T00:36:18.962 INFO:teuthology.orchestra.run.smithi064.stdout:4305 writesame 0x50ab3c3 thru 0x50b7f92 data_size 0x18(0xcbd0 bytes) 2024-04-24T00:36:18.965 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:18.959+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:18.969 INFO:teuthology.orchestra.run.smithi064.stdout:4306 read 0x304297d thru 0x3042a66 (0xea bytes) 2024-04-24T00:36:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:4307 read 0x114576c thru 0x1145979 (0x20e bytes) 2024-04-24T00:36:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:4309 read 0xbc6a308 thru 0xbc70916 (0x660f bytes) 2024-04-24T00:36:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:4312 punch from 0x226ed3b to 0x227d499, (0xe75e bytes) 2024-04-24T00:36:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:4313 read 0xb69f9a6 thru 0xb6a09d0 (0x102b bytes) 2024-04-24T00:36:18.980 INFO:teuthology.orchestra.run.smithi064.stdout:4314 trunc from 0xc27679e to 0x5b17069 2024-04-24T00:36:19.021 INFO:teuthology.orchestra.run.smithi064.stdout:4315 writesame 0xa2ffcf1 thru 0xa30e4f9 data_size 0x1(0xe809 bytes) 2024-04-24T00:36:19.021 INFO:teuthology.orchestra.run.smithi064.stdout:4316 punch from 0x1490fcf to 0x149a01d, (0x904e bytes) 2024-04-24T00:36:19.021 INFO:teuthology.orchestra.run.smithi064.stdout:4317 write 0x6deb604 thru 0x6df7ee8 (0xc8e5 bytes) 2024-04-24T00:36:19.038 INFO:teuthology.orchestra.run.smithi064.stdout:4319 writesame 0x2c7277d thru 0x2c73f60 data_size 0x16(0x17e4 bytes) 2024-04-24T00:36:19.047 INFO:teuthology.orchestra.run.smithi064.stdout:4321 write 0x66d85eb thru 0x66df9c5 (0x73db bytes) 2024-04-24T00:36:19.053 INFO:teuthology.orchestra.run.smithi064.stdout:4322 write 0x117e46f thru 0x118afcd (0xcb5f bytes) 2024-04-24T00:36:19.064 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:19.064 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:19.065 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:19.060+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:19.066 INFO:teuthology.orchestra.run.smithi064.stdout:4323 read 0x697aa5a thru 0x698a329 (0xf8d0 bytes) 2024-04-24T00:36:19.066 INFO:teuthology.orchestra.run.smithi064.stdout:4325 read 0x3e9c831 thru 0x3ea6bdd (0xa3ad bytes) 2024-04-24T00:36:19.067 INFO:teuthology.orchestra.run.smithi064.stdout:4329 write 0x44def41 thru 0x44e3b79 (0x4c39 bytes) 2024-04-24T00:36:19.072 INFO:teuthology.orchestra.run.smithi064.stdout:4331 write 0x4ff58bc thru 0x4ffb0c6 (0x580b bytes) 2024-04-24T00:36:19.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:19.093 INFO:teuthology.orchestra.run.smithi064.stdout:4332 trunc from 0xa30e4fa to 0x519df32 2024-04-24T00:36:19.121 INFO:teuthology.orchestra.run.smithi064.stdout:4333 punch from 0x2eb2493 to 0x2eb9b49, (0x76b6 bytes) 2024-04-24T00:36:19.121 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:19.121 INFO:teuthology.orchestra.run.smithi064.stdout:4335 read 0x1baf8ea thru 0x1bb64a9 (0x6bc0 bytes) 2024-04-24T00:36:19.128 INFO:teuthology.orchestra.run.smithi064.stdout:4337 trunc from 0x519df32 to 0x21656ad 2024-04-24T00:36:19.150 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:19.166 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:19.164+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:19.183 INFO:teuthology.orchestra.run.smithi064.stdout:4339 write 0xc8ff563 thru 0xc90dfa0 (0xea3e bytes) 2024-04-24T00:36:19.191 INFO:teuthology.orchestra.run.smithi064.stdout:4340 write 0x2c14fa9 thru 0x2c1f287 (0xa2df bytes) 2024-04-24T00:36:19.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:19.225 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:19.225 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:19.229 INFO:teuthology.orchestra.run.smithi064.stdout:4342 trunc from 0xc90dfa1 to 0x1cf719c 2024-04-24T00:36:19.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:19.269 INFO:teuthology.orchestra.run.smithi064.stdout:4344 punch from 0xb8dff9 to 0xb932a4, (0x52ab bytes) 2024-04-24T00:36:19.269 INFO:teuthology.orchestra.run.smithi064.stdout:4345 write 0xe97a83 thru 0xe9b9bd (0x3f3b bytes) 2024-04-24T00:36:19.270 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:19.268+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:19.278 INFO:teuthology.orchestra.run.smithi064.stdout:4347 punch from 0x2f4555 to 0x300fb5, (0xca60 bytes) 2024-04-24T00:36:19.289 INFO:teuthology.orchestra.run.smithi064.stdout:4348 writesame 0x33e55a9 thru 0x33eccdb data_size 0x11(0x7733 bytes) 2024-04-24T00:36:19.313 INFO:teuthology.orchestra.run.smithi064.stdout:4349 write 0xdec83a9 thru 0xded1593 (0x91eb bytes) 2024-04-24T00:36:19.320 INFO:teuthology.orchestra.run.smithi064.stdout:4350 write 0xc8a534 thru 0xc92d4d (0x881a bytes) 2024-04-24T00:36:19.345 INFO:teuthology.orchestra.run.smithi064.stdout:4351 trunc from 0xded1594 to 0xbe674e 2024-04-24T00:36:19.367 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:19.364+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:19.385 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:19.385 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:19.405 INFO:teuthology.orchestra.run.smithi064.stdout:4353 writesame 0x99b5789 thru 0x99bfa3b data_size 0x1(0xa2b3 bytes) 2024-04-24T00:36:19.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.97287970399999, 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:36:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:36:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:36:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:36:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:36:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.91736821800001, 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:36:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:36:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.816791570000007, 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:36:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.775226615999998, 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:36:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.211495053999997, 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:36:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:36:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.708521656999999, 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:36:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.470207424000002, 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435267001, 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:36:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435246128000003, 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:36:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435222748999998, 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:36:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435201206999999, 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435179687000002, 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:36:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:36:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435133530000002, 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435109410999999, 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:36:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435086007000002, 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435058234000003, 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:36:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:36:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:36:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435036820999997, 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:36:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:36:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:36:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:36:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:36:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:36:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:36:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:36:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:36:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.435016230000002, 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.218945988000002, 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:36:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:36:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:36:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.190975084000002, 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:36:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:36:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:36:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:36:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:36:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:36:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:36:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:36:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:36:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:36:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:36:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:36:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:19.512 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:19.513 INFO:teuthology.orchestra.run.smithi064.stdout:4354 writesame 0xd84cc67 thru 0xd85c843 data_size 0x3d(0xfbdd bytes) 2024-04-24T00:36:19.514 INFO:teuthology.orchestra.run.smithi064.stdout:4356 trunc from 0xd85c844 to 0x6bba5c1 2024-04-24T00:36:19.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:19.516 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:19.516 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:19.517 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:19.517 INFO:teuthology.orchestra.run.smithi064.stdout:4358 write 0x33a2c57 thru 0x33a9ac6 (0x6e70 bytes) 2024-04-24T00:36:19.517 INFO:teuthology.orchestra.run.smithi064.stdout:4359 writesame 0x1846004 thru 0x184c095 data_size 0x2f(0x6092 bytes) 2024-04-24T00:36:19.518 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:19.518 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:19.519 INFO:teuthology.orchestra.run.smithi064.stdout:4360 write 0x7768cf2 thru 0x7769541 (0x850 bytes) 2024-04-24T00:36:19.519 INFO:teuthology.orchestra.run.smithi064.stdout:4361 trunc from 0x7769542 to 0x7044f9c 2024-04-24T00:36:19.519 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:19.520 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:4363 read 0x4a0cc3 thru 0x4ad65d (0xc99b bytes) 2024-04-24T00:36:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:19.522 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:19.523 INFO:teuthology.orchestra.run.smithi064.stdout:4365 write 0xd118d20 thru 0xd121938 (0x8c19 bytes) 2024-04-24T00:36:19.523 INFO:teuthology.orchestra.run.smithi064.stdout:4366 writesame 0x9eb8476 thru 0x9ec13b4 data_size 0x1(0x8f3f bytes) 2024-04-24T00:36:19.523 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:19.512+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:19.524 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:19.540 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:19.614 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:19.612+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:19.641 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:19.642 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:19.646 INFO:teuthology.orchestra.run.smithi064.stdout:4368 writesame 0x7ee44c7 thru 0x7ee6e1e data_size 0x1c(0x2958 bytes) 2024-04-24T00:36:19.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:19.665 INFO:teuthology.orchestra.run.smithi064.stdout:4369 writesame 0xd79759b thru 0xd79cd2f data_size 0x1(0x5795 bytes) 2024-04-24T00:36:19.665 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:19.665 INFO:teuthology.orchestra.run.smithi064.stdout:4374 write 0x324c29a thru 0x325a1be (0xdf25 bytes) 2024-04-24T00:36:19.690 INFO:teuthology.orchestra.run.smithi064.stdout:4375 trunc from 0xd79cd30 to 0x3e86505 2024-04-24T00:36:19.716 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:19.712+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:19.727 INFO:teuthology.orchestra.run.smithi064.stdout:4376 write 0x7a48a8a thru 0x7a4c602 (0x3b79 bytes) 2024-04-24T00:36:19.733 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:19.742 INFO:teuthology.orchestra.run.smithi064.stdout:4380 writesame 0x7b8fc11 thru 0x7b948f0 data_size 0x29(0x4ce0 bytes) 2024-04-24T00:36:19.747 INFO:teuthology.orchestra.run.smithi064.stdout:4382 trunc from 0x7b948f1 to 0x6c829d3 2024-04-24T00:36:19.777 INFO:teuthology.orchestra.run.smithi064.stdout:4383 write 0x6e73b59 thru 0x6e76f91 (0x3439 bytes) 2024-04-24T00:36:19.804 INFO:teuthology.orchestra.run.smithi064.stdout:4384 write 0xbbb4e9b thru 0xbbc1603 (0xc769 bytes) 2024-04-24T00:36:19.806 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:36:19.807 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:19.807 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:19.810 INFO:teuthology.orchestra.run.smithi064.stdout:4385 punch from 0xad8ccde to 0xad990cc, (0xc3ee bytes) 2024-04-24T00:36:19.817 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:19.811+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:19.828 INFO:teuthology.orchestra.run.smithi064.stdout:4386 trunc from 0xbbc1604 to 0x1275812 2024-04-24T00:36:19.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:19.863 INFO:teuthology.orchestra.run.smithi064.stdout:4388 read 0xa7c1e6 thru 0xa8279f (0x65ba bytes) 2024-04-24T00:36:19.864 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:19.864 INFO:teuthology.orchestra.run.smithi064.stdout:4390 read 0x605a0d thru 0x60fc99 (0xa28d bytes) 2024-04-24T00:36:19.865 INFO:teuthology.orchestra.run.smithi064.stdout:4391 read 0xe3d870 thru 0xe49f8d (0xc71e bytes) 2024-04-24T00:36:19.865 INFO:teuthology.orchestra.run.smithi064.stdout:4392 punch from 0xf9e0af to 0xfa5dae, (0x7cff bytes) 2024-04-24T00:36:19.878 INFO:teuthology.orchestra.run.smithi064.stdout:4393 writesame 0x442ec57 thru 0x443816b data_size 0x11(0x9515 bytes) 2024-04-24T00:36:19.890 INFO:teuthology.orchestra.run.smithi064.stdout:4395 writesame 0x6020a47 thru 0x6024417 data_size 0x13(0x39d1 bytes) 2024-04-24T00:36:19.890 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:19.896 INFO:teuthology.orchestra.run.smithi064.stdout:4398 trunc from 0x6024418 to 0x9eb770c 2024-04-24T00:36:19.918 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:19.916+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:19.919 INFO:teuthology.orchestra.run.smithi064.stdout:4400 write 0xdb068e9 thru 0xdb0f4a5 (0x8bbd bytes) 2024-04-24T00:36:19.926 INFO:teuthology.orchestra.run.smithi064.stdout:4403 read 0x3f4d53c thru 0x3f55ee4 (0x89a9 bytes) 2024-04-24T00:36:19.928 INFO:teuthology.orchestra.run.smithi064.stdout:4404 trunc from 0xdb0f4a6 to 0xd2c9902 2024-04-24T00:36:19.946 INFO:teuthology.orchestra.run.smithi064.stdout:4405 punch from 0x65f62c8 to 0x65f70a8, (0xde0 bytes) 2024-04-24T00:36:19.946 INFO:teuthology.orchestra.run.smithi064.stdout:4408 read 0x8fa4f33 thru 0x8fabac9 (0x6b97 bytes) 2024-04-24T00:36:19.946 INFO:teuthology.orchestra.run.smithi064.stdout:4411 write 0x443b0f9 thru 0x44487a4 (0xd6ac bytes) 2024-04-24T00:36:19.973 INFO:teuthology.orchestra.run.smithi064.stdout:4412 writesame 0xe6cfd0e thru 0xe6d78f2 data_size 0x17(0x7be5 bytes) 2024-04-24T00:36:19.973 INFO:teuthology.orchestra.run.smithi064.stdout:4413 read 0xcd7e512 thru 0xcd8368a (0x5179 bytes) 2024-04-24T00:36:19.973 INFO:teuthology.orchestra.run.smithi064.stdout:4414 writesame 0xe43b03d thru 0xe444976 data_size 0xb(0x993a bytes) 2024-04-24T00:36:19.988 INFO:teuthology.orchestra.run.smithi064.stdout:4415 read 0x5d06088 thru 0x5d0b51c (0x5495 bytes) 2024-04-24T00:36:19.989 INFO:teuthology.orchestra.run.smithi064.stdout:4416 write 0x3b32f92 thru 0x3b3a37e (0x73ed bytes) 2024-04-24T00:36:19.996 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 112.711057074, 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:36:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:36:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head, version: 45'133, data_included: [0~131072,196158~37038,262144~131072,524288~131072,786432~262144,1258308~52412,1441792~131072,3014656~131072,3267061~6628,6160384~34994], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head@45'133, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 6195378, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.068845682999999, 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011957290000000001, 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922063+0000", 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.306e-06 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923213+0000", 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011495310000000001 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923258+0000", 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4747999999999999e-05 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923284+0000", 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5873000000000001e-05 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923436+0000", 2024-04-24T00:36:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015226500000000001 2024-04-24T00:36:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923452+0000", 2024-04-24T00:36:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6022e-05 2024-04-24T00:36:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.934018+0000", 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010566545 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head, version: 45'137, data_included: [0~131072,262144~131072,524288~131072,786432~199751,1179648~1552,1441792~3851,3014656~131072,5775043~64169,6160384~131072], data_size: 924683, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head@45'137, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8781824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.040020961000003, 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011581296, 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950888+0000", 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951161+0000", 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027280499999999999 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951196+0000", 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123999999999998e-05 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951224+0000", 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8045e-05 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951356+0000", 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013187799999999999 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951376+0000", 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.003e-05 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.962467+0000", 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011091474 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.834686146000003, 2024-04-24T00:36:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head, version: 80'319, data_included: [0~393216,524288~131072,1179648~524288], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head@80'319, size: 13631488, copy_subset: [(0, 13631488)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.821932052999998, 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089307889999999997, 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168976+0000", 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169620+0000", 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064358299999999998 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169650+0000", 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0111e-05 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169671+0000", 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1231000000000001e-05 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169789+0000", 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118368 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169803+0000", 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3779e-05 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.177905+0000", 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081024800000000004 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.809785107000003, 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.793040949000002, 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.702891956000002, 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.295735276999999, 2024-04-24T00:36:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.285348580000001, 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.276405291, 2024-04-24T00:36:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:90104 3.18 3:193b8512:::rbd_data.115e56aa02fc.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write 1644318~44706] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.880272397999999, 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084175689999999997, 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 90104 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110634+0000", 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110636+0000", 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0229999999999999e-06 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110706+0000", 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9392000000000002e-05 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110734+0000", 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8538999999999999e-05 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110747+0000", 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860000000000001e-05 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.110850+0000", 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102988 2024-04-24T00:36:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111107+0000", 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025716600000000002 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.111346+0000", 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000238373 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.117684+0000", 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063379949999999999 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118724+0000", 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001040685 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.118914+0000", 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018984699999999999 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.119052+0000", 2024-04-24T00:36:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137701 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPushReply(3.12 128/127 [PushReplyOp(3:4e045fb2:::rbd_data.115e6f07caf6.0000000000000007:head)])", 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.613083028999998, 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089128709999999993, 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377824+0000", 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377827+0000", 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5870000000000001e-06 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377852+0000", 2024-04-24T00:36:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5083e-05 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377890+0000", 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7279e-05 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.377914+0000", 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4496000000000001e-05 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.378154+0000", 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024028 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:52.386737+0000", 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085821460000000006 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:92353 3.0 3:02b978df:::rbd_data.115ee57427d8.0000000000000008:head [truncate 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.237442741000001, 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091800189999999993, 2024-04-24T00:36:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 92353 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753467+0000", 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7259999999999998e-06 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753478+0000", 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1456e-05 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753503+0000", 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5015999999999999e-05 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753516+0000", 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.259e-05 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753615+0000", 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9104999999999994e-05 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753624+0000", 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9309999999999997e-06 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755534+0000", 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019101700000000001 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755620+0000", 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5736999999999996e-05 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755756+0000", 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013602800000000001 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.756642+0000", 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088663200000000005 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762416+0000", 2024-04-24T00:36:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057733849999999998 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762534+0000", 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118594 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762644+0000", 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010964900000000001 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:94358 3.0 3:06002a2d:::rbd_data.115eb3626b1c.000000000000002f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2512793~56897] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2518429290000004, 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010252809999999999, 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 94358 2024-04-24T00:36:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739067+0000", 2024-04-24T00:36:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9299999999999999e-06 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739156+0000", 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9148999999999996e-05 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739189+0000", 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2772999999999999e-05 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739205+0000", 2024-04-24T00:36:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6722999999999999e-05 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739285+0000", 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9729999999999997e-05 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739348+0000", 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3309e-05 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739445+0000", 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6378000000000005e-05 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.742520+0000", 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030747819999999999 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749133+0000", 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066130659999999999 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749238+0000", 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010506299999999999 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749317+0000", 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8906999999999995e-05 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105170 3.16 e152/112 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head v 152'511, mlcod=150'509)", 2024-04-24T00:36:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.3017116269999995, 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022303394000000001, 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689196+0000", 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7399999999999991e-07 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692342+0000", 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031461710000000001 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692378+0000", 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5707999999999998e-05 2024-04-24T00:36:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692396+0000", 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8110000000000001e-05 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692483+0000", 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6565999999999998e-05 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692513+0000", 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0151999999999999e-05 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711444+0000", 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018930827000000001 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711499+0000", 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4886000000000003e-05 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105172 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [write 0~917504 [fadvise_sequential+fadvise_nocache],zero 917504~393216,write 1310720~131072 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.3010723730000002, 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024973104999999999, 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105172 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689837+0000", 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-06 2024-04-24T00:36:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693186+0000", 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033495899999999999 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693214+0000", 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7616000000000001e-05 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693229+0000", 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5424999999999999e-05 2024-04-24T00:36:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693350+0000", 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120711 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693534+0000", 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018363200000000001 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693878+0000", 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000343958 2024-04-24T00:36:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.712994+0000", 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019116371 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714688+0000", 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016935539999999999 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714754+0000", 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6297000000000006e-05 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714807+0000", 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3522e-05 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105169 3.1f e152/59 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head v 152'697, mlcod=152'693)", 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2983929419999996, 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041348306000000001, 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692515+0000", 2024-04-24T00:36:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699647+0000", 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071313169999999999 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699676+0000", 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9876999999999999e-05 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699693+0000", 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6973000000000002e-05 2024-04-24T00:36:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699779+0000", 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5169999999999999e-05 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699805+0000", 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6120000000000001e-05 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733817+0000", 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034012112999999997 2024-04-24T00:36:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733862+0000", 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5252e-05 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105196 3.0 3:00473128:::rbd_data.115edc839ccf.00000000000001ff:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.704009+0000", 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2868976170000002, 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085699269999999998, 2024-04-24T00:36:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105196 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.704009+0000", 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.704009+0000", 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.704010+0000", 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.704017+0000", 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9020000000000002e-06 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.704035+0000", 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7917000000000001e-05 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.704048+0000", 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.273e-05 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.712478+0000", 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0084299930000000002 2024-04-24T00:36:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.712579+0000", 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101065 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106528 3.12 3:4eea8178:::rbd_data.115ec2448e7a.000000000000000c:head [write 2409114~57125] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:19.664438+0000", 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.32646841300000001, 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089572520000000006, 2024-04-24T00:36:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106528 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664438+0000", 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664438+0000", 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664440+0000", 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.268e-06 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664464+0000", 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4570999999999999e-05 2024-04-24T00:36:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664478+0000", 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4209000000000001e-05 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664486+0000", 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1389999999999998e-06 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664542+0000", 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6406000000000002e-05 2024-04-24T00:36:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664602+0000", 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0387999999999998e-05 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664689+0000", 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7026000000000003e-05 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.667625+0000", 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029353750000000001 2024-04-24T00:36:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.673277+0000", 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056522150000000004 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.673326+0000", 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8681999999999999e-05 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.673396+0000", 2024-04-24T00:36:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9973000000000002e-05 2024-04-24T00:36:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.095 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:20.096 INFO:teuthology.orchestra.run.smithi064.stdout:4418 trunc from 0xe6d78f3 to 0xd78afc8 2024-04-24T00:36:20.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:20.098 INFO:teuthology.orchestra.run.smithi064.stdout:4419 writesame 0xe64c8ec thru 0xe650757 data_size 0x55(0x3e6c bytes) 2024-04-24T00:36:20.098 INFO:teuthology.orchestra.run.smithi064.stdout:4421 write 0xa19c808 thru 0xa1a092f (0x4128 bytes) 2024-04-24T00:36:20.098 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:4424 read 0x235786 thru 0x244102 (0xe97d bytes) 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:4425 writesame 0x67684a1 thru 0x6774e3f data_size 0x9b(0xc99f bytes) 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:4426 write 0xfd5333 thru 0xfd6666 (0x1334 bytes) 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:4427 write 0x7aacd72 thru 0x7aae419 (0x16a8 bytes) 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:4429 write 0x486f02b thru 0x487267a (0x3650 bytes) 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:4431 read 0x2ab2af4 thru 0x2ac0422 (0xd92f bytes) 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:4433 write 0x42994af thru 0x42a5a3b (0xc58d bytes) 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:20.099 INFO:teuthology.orchestra.run.smithi064.stdout:4435 punch from 0xb0243ce to 0xb02f296, (0xaec8 bytes) 2024-04-24T00:36:20.099 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:20.092+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:20.110 INFO:teuthology.orchestra.run.smithi064.stdout:4436 trunc from 0xe650758 to 0x43e0127 2024-04-24T00:36:20.165 INFO:teuthology.orchestra.run.smithi064.stdout:4438 write 0xad7423a thru 0xad829ea (0xe7b1 bytes) 2024-04-24T00:36:20.184 INFO:teuthology.orchestra.run.smithi064.stdout:4439 trunc from 0xad829eb to 0x6561993 2024-04-24T00:36:20.197 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:20.195+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:20.211 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:20.211 INFO:teuthology.orchestra.run.smithi064.stdout:4441 punch from 0x618a2f9 to 0x6194bb5, (0xa8bc bytes) 2024-04-24T00:36:20.211 INFO:teuthology.orchestra.run.smithi064.stdout:4442 read 0x155d83b thru 0x1566e96 (0x965c bytes) 2024-04-24T00:36:20.225 INFO:teuthology.orchestra.run.smithi064.stdout:4443 trunc from 0x6561993 to 0xc4dae9e 2024-04-24T00:36:20.238 INFO:teuthology.orchestra.run.smithi064.stdout:4446 writesame 0x4cdf14d thru 0x4cece57 data_size 0x1(0xdd0b bytes) 2024-04-24T00:36:20.261 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:20.261 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:20.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:20.334 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:20.332+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:20.365 INFO:teuthology.orchestra.run.smithi064.stdout:4447 read 0x53f7d25 thru 0x53fd744 (0x5a20 bytes) 2024-04-24T00:36:20.381 INFO:teuthology.orchestra.run.smithi064.stdout:4448 trunc from 0xc4dae9e to 0x2163115 2024-04-24T00:36:20.398 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:20.395+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:20.402 INFO:teuthology.orchestra.run.smithi064.stdout:4449 read 0x70623c thru 0x70e5ba (0x837f bytes) 2024-04-24T00:36:20.403 INFO:teuthology.orchestra.run.smithi064.stdout:4452 punch from 0xb423a4 to 0xb4da82, (0xb6de bytes) 2024-04-24T00:36:20.414 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:20.414 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:20.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:20.432 INFO:teuthology.orchestra.run.smithi064.stdout:4453 write 0x95237ab thru 0x95279cc (0x4222 bytes) 2024-04-24T00:36:20.447 INFO:teuthology.orchestra.run.smithi064.stdout:4456 trunc from 0x95279cd to 0x442df42 2024-04-24T00:36:20.468 INFO:teuthology.orchestra.run.smithi064.stdout:4458 read 0xeff4b3 thru 0xf01ef1 (0x2a3f bytes) 2024-04-24T00:36:20.469 INFO:teuthology.orchestra.run.smithi064.stdout:4459 read 0x3a9978c thru 0x3a9d572 (0x3de7 bytes) 2024-04-24T00:36:20.470 INFO:teuthology.orchestra.run.smithi064.stdout:4461 read 0x1ec929f thru 0x1ed3746 (0xa4a8 bytes) 2024-04-24T00:36:20.482 INFO:teuthology.orchestra.run.smithi064.stdout:4462 write 0x62b6882 thru 0x62ba631 (0x3db0 bytes) 2024-04-24T00:36:20.489 INFO:teuthology.orchestra.run.smithi064.stdout:4463 punch from 0x244c97c to 0x245499e, (0x8022 bytes) 2024-04-24T00:36:20.499 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:20.496+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:20.507 INFO:teuthology.orchestra.run.smithi064.stdout:4464 write 0x964f675 thru 0x96504ee (0xe7a bytes) 2024-04-24T00:36:20.513 INFO:teuthology.orchestra.run.smithi064.stdout:4466 write 0x3920c5 thru 0x39e3fe (0xc33a bytes) 2024-04-24T00:36:20.522 INFO:teuthology.orchestra.run.smithi064.stdout:4468 writesame 0x1de449d thru 0x1de8b58 data_size 0x9(0x46bc bytes) 2024-04-24T00:36:20.533 INFO:teuthology.orchestra.run.smithi064.stdout:4469 writesame 0x375e0d0 thru 0x376ccb9 data_size 0x1(0xebea bytes) 2024-04-24T00:36:20.583 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.366913563, 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.589 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.366562648, 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:20.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.591 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:20.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:20.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:20.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.359674328, 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:20.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.600 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.101392701999998, 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:20.606 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.091116286000002, 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:20.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.629165483000001, 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.616757217, 2024-04-24T00:36:20.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.616725463000002, 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:20.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.613192967000003, 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:20.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.603131462, 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.389188554999997, 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:20.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:20.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:20.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.379176610999998, 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.105797748999997, 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.642 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:20.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:20.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:20.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:20.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.089129300000003, 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:20.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:20.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.903231517000002, 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.652 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:20.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:20.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:20.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:20.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.21947316, 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:20.657 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:20.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:20.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.197256161999999, 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:20.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.036034881999999, 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.667 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:20.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:20.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8926863239999996, 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:20.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:20.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:36:20.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8800613640000003, 2024-04-24T00:36:20.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:20.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:20.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:20.684 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:20.684 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:20.679+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:20.710 INFO:teuthology.orchestra.run.smithi064.stdout:4471 read 0x344c6d4 thru 0x344fe47 (0x3774 bytes) 2024-04-24T00:36:20.711 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:20.711 INFO:teuthology.orchestra.run.smithi064.stdout:4476 punch from 0x33dbde0 to 0x33e1a53, (0x5c73 bytes) 2024-04-24T00:36:20.711 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:20.730 INFO:teuthology.orchestra.run.smithi064.stdout:4478 write 0xb9a2a88 thru 0xb9a9ea2 (0x741b bytes) 2024-04-24T00:36:20.740 INFO:teuthology.orchestra.run.smithi064.stdout:4479 trunc from 0xb9a9ea3 to 0xcd7ec6e 2024-04-24T00:36:20.755 INFO:teuthology.orchestra.run.smithi064.stdout:4480 clone 29 order 19 su 32768 sc 14 2024-04-24T00:36:20.784 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:20.779+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:20.818 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:20.819 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:20.819 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:20.819 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:20.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:20.926 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:20.923+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:20.986 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:20.983+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:21.005 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:21.005 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:21.006 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:21.006 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:21.006 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:21.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:21.087 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:21.084+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:21.187 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.96795521200001, 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:21.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:21.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.546169518, 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:21.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:21.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:21.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:21.199 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.404041952, 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:21.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:21.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:21.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.266270169999999, 2024-04-24T00:36:21.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:21.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:21.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:21.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.212 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.228787852000004, 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:21.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:21.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:21.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:21.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:21.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:21.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.199458890000002, 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:36:21.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:36:21.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:21.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.002196986999998, 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.225 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:21.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:21.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.566441988000001, 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.231 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:21.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.556583951999997, 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.237 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:21.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:21.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:21.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:21.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:21.242 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.536234344, 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:21.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:21.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.502189008999999, 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.248 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:21.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:21.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.499879172999997, 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:21.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.491310374000001, 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:21.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:21.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.552161281, 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:21.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:21.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.271 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.662771285, 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:21.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:21.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:21.277 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.619295488000001, 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:21.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:21.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.645322186, 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:21.281 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:21.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:21.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.626180283, 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.290 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.66689273, 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.291 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:21.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:21.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:21.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.6328903079999999, 2024-04-24T00:36:21.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:21.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:21.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:21.300 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:21.300 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:21.183+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:21.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:21.346 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:21.343+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:21.404 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:21.399+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:21.472 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:21.472 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:21.472 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:21.472 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:21.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:21.530 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:21.527+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:21.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:21.605 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:21.599+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:21.610 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [0, 7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:21.610 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:21.610 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-24T00:36:21.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-04-24T00:36:21.650 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:21.651 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:21.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:21.706 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:21.703+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:21.716 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:21.716 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:21.732 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:21.733 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:21.751 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:21.751 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:21.767 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:21.768 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:21.805 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:21.805 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:21.806 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:21.804+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:21.834 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:21.834 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:21.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:36:21.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:21.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:21.853 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:21.918 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:21.916+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:21.955 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:36:21.952+0000 7f841a5f8640 -1 mon.a@0(leader).osd e156 definitely_dead 0 2024-04-24T00:36:22.008 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:22.004+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:22.026 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:22.026 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:22.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:22.110 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:22.108+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:22.210 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:22.208+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:22.228 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:22.228 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:22.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:22.311 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:22.307+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:22.384 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:22.385 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:22.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:22.442 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:22.439+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:22.514 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:22.512+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:22.561 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:22.562 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:22.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:22.621 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:22.619+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:22.697 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:22.698 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:22.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:22.716 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:22.712+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:22.817 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:22.815+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:22.840 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.0. 2024-04-24T00:36:22.840 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:36:22.873 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-24T00:36:22.890 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:22.890 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:22.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:22.930 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:22.928+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:23.018 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:23.015+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:23.047 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:23.047 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:23.061 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:23.119 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:23.115+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:23.188 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:23.188 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:23.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:23.250 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:23.248+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:23.321 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:23.320+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:23.322 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:23.322 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:23.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:23.422 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:23.419+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:23.503 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:23.503 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:23.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:23.562 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:23.560+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:23.624 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:23.619+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:23.640 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:23.640 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:23.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:23.725 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:23.719+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:23.825 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:23.826 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:23.826 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:23.823+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:23.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:23.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:23.927 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:23.923+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:23.962 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:23.963 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:23.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:24.028 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:24.024+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:24.031 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:24.032 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:24.049 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:24.049 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:24.066 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:24.066 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:24.084 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:24.084 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:24.103 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:24.103 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:24.104 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:24.104 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:24.105 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:24.105 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:24.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:24.127 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:24.127 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:24.129 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:24.124+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:24.151 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:24.151 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:24.173 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:24.173 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:24.231 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:24.227+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:24.250 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:24.250 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:21610 3.9 e52/21 3:91b69534:::rbd_data.115eeb9272a5.0000000000000010:head v 52'92, mlcod=52'90)", 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.96566765999999, 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091681190000000006, 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279850+0000", 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.279852+0000", 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.007e-06 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280079+0000", 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022780399999999999 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280109+0000", 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9984999999999998e-05 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280133+0000", 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3876e-05 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280266+0000", 2024-04-24T00:36:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132361 2024-04-24T00:36:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.280307+0000", 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1591999999999998e-05 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.288901+0000", 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0085933320000000004 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.289018+0000", 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117162 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head, version: 45'133, data_included: [0~131072,196158~37038,262144~131072,524288~131072,786432~262144,1258308~52412,1441792~131072,3014656~131072,3267061~6628,6160384~34994], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6c4aeebf:::rbd_data.115e8003c269.0000000000000003:head@45'133, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 6195378, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.323456268999998, 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011957290000000001, 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922061+0000", 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.922063+0000", 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.306e-06 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923213+0000", 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011495310000000001 2024-04-24T00:36:24.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923258+0000", 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4747999999999999e-05 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923284+0000", 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5873000000000001e-05 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923436+0000", 2024-04-24T00:36:24.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015226500000000001 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.923452+0000", 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6022e-05 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.934018+0000", 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010566545 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head, version: 45'137, data_included: [0~131072,262144~131072,524288~131072,786432~199751,1179648~1552,1441792~3851,3014656~131072,5775043~64169,6160384~131072], data_size: 924683, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6a5f13d0:::rbd_data.115e8003c269.0000000000000004:head@45'137, size: 8781824, copy_subset: [(0, 8781824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 8781824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.294631547000002, 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011581296, 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950886+0000", 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.950888+0000", 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951161+0000", 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027280499999999999 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951196+0000", 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5123999999999998e-05 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951224+0000", 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8045e-05 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951356+0000", 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013187799999999999 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.951376+0000", 2024-04-24T00:36:24.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.003e-05 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.962467+0000", 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011091474 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:24.264 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.089296732000001, 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:24.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:24.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(3.16 111/110 [PushOp(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head, version: 80'319, data_included: [0~393216,524288~131072,1179648~524288], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:6d3b2812:::rbd_data.115ec7ed39de.0000000000000004:head@80'319, size: 13631488, copy_subset: [(0, 13631488)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.076542639000003, 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089307889999999997, 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168975+0000", 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.168976+0000", 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2024-04-24T00:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169620+0000", 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064358299999999998 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169650+0000", 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0111e-05 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169671+0000", 2024-04-24T00:36:24.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1231000000000001e-05 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169789+0000", 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118368 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.169803+0000", 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3779e-05 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.177905+0000", 2024-04-24T00:36:24.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081024800000000004 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.064395693000002, 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.047651535, 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:24.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.957502542, 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:24.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:24.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:24.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.550345862999997, 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.287 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:24.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:24.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:24.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.539959166000003, 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:24.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:24.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:24.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.531015877000002, 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:24.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:24.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:24.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:92353 3.0 3:02b978df:::rbd_data.115ee57427d8.0000000000000008:head [truncate 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.492053327000001, 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091800189999999993, 2024-04-24T00:36:24.302 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 92353 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753464+0000", 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753467+0000", 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7259999999999998e-06 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753478+0000", 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1456e-05 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753503+0000", 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5015999999999999e-05 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753516+0000", 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.259e-05 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753615+0000", 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9104999999999994e-05 2024-04-24T00:36:24.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.753624+0000", 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9309999999999997e-06 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755534+0000", 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019101700000000001 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755620+0000", 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5736999999999996e-05 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.755756+0000", 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013602800000000001 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.756642+0000", 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088663200000000005 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762416+0000", 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057733849999999998 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762534+0000", 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118594 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:06.762644+0000", 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010964900000000001 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:94358 3.0 3:06002a2d:::rbd_data.115eb3626b1c.000000000000002f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2512793~56897] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.506453515, 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010252809999999999, 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 94358 2024-04-24T00:36:24.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739064+0000", 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739067+0000", 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9299999999999999e-06 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739156+0000", 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9148999999999996e-05 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739189+0000", 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2772999999999999e-05 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739205+0000", 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6722999999999999e-05 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739285+0000", 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9729999999999997e-05 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739348+0000", 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3309e-05 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.739445+0000", 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6378000000000005e-05 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.742520+0000", 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030747819999999999 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749133+0000", 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066130659999999999 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749238+0000", 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010506299999999999 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:10.749317+0000", 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8906999999999995e-05 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.314 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105170 3.16 e152/112 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head v 152'511, mlcod=150'509)", 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.5563222129999996, 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022303394000000001, 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689196+0000", 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7399999999999991e-07 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692342+0000", 2024-04-24T00:36:24.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031461710000000001 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692378+0000", 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5707999999999998e-05 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692396+0000", 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8110000000000001e-05 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692483+0000", 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6565999999999998e-05 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692513+0000", 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0151999999999999e-05 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711444+0000", 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018930827000000001 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711499+0000", 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4886000000000003e-05 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105172 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [write 0~917504 [fadvise_sequential+fadvise_nocache],zero 917504~393216,write 1310720~131072 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.5556829590000003, 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024973104999999999, 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105172 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689837+0000", 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-06 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693186+0000", 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033495899999999999 2024-04-24T00:36:24.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693214+0000", 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7616000000000001e-05 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693229+0000", 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5424999999999999e-05 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693350+0000", 2024-04-24T00:36:24.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120711 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693534+0000", 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018363200000000001 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693878+0000", 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000343958 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.712994+0000", 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019116371 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714688+0000", 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016935539999999999 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714754+0000", 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6297000000000006e-05 2024-04-24T00:36:24.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714807+0000", 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3522e-05 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105169 3.1f e152/59 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head v 152'697, mlcod=152'693)", 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.5530035279999996, 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041348306000000001, 2024-04-24T00:36:24.325 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692515+0000", 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699647+0000", 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071313169999999999 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699676+0000", 2024-04-24T00:36:24.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9876999999999999e-05 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699693+0000", 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6973000000000002e-05 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699779+0000", 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5169999999999999e-05 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699805+0000", 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6120000000000001e-05 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733817+0000", 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034012112999999997 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733862+0000", 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5252e-05 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106528 3.12 3:4eea8178:::rbd_data.115ec2448e7a.000000000000000c:head [write 2409114~57125] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:19.664438+0000", 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5810789989999998, 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0089572520000000006, 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106528 2024-04-24T00:36:24.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664438+0000", 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664438+0000", 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664440+0000", 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.268e-06 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664464+0000", 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4570999999999999e-05 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664478+0000", 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4209000000000001e-05 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664486+0000", 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1389999999999998e-06 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664542+0000", 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6406000000000002e-05 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664602+0000", 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0387999999999998e-05 2024-04-24T00:36:24.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.664689+0000", 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7026000000000003e-05 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.667625+0000", 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029353750000000001 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.673277+0000", 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056522150000000004 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.673326+0000", 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8681999999999999e-05 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:19.673396+0000", 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9973000000000002e-05 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.0048190229999996, 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.336 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:36:24.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:36:24.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9750515420000001, 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:36:24.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:36:24.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106874 3.18 3:1ecbe779:::rbd_header.115ec2448e7a:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.816089+0000", 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.4294285360000001, 2024-04-24T00:36:24.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012895386999999999, 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106874 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816089+0000", 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816089+0000", 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816092+0000", 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.05e-06 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816107+0000", 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4983e-05 2024-04-24T00:36:24.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816149+0000", 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1712000000000003e-05 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816174+0000", 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5612000000000001e-05 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816414+0000", 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023948599999999999 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816426+0000", 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2244e-05 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.825743+0000", 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093174759999999999 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.825895+0000", 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015211599999999999 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.826543+0000", 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064714000000000004 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.827851+0000", 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013083159999999999 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.828704+0000", 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085295399999999995 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.828877+0000", 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017332099999999999 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.828984+0000", 2024-04-24T00:36:24.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106977 2024-04-24T00:36:24.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.353 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:24.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:24.355 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:24.351+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:24.456 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:24.452+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:24.485 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:24.485 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:24.485 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:24.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:24.586 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:24.584+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:24.642 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-04-24T00:36:24.657 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:24.655+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:24.737 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:24.737 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:24.737 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:24.737 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:24.737 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:24.758 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:24.756+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:24.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:24.859 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:24.856+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:24.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:36:24.959 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 131.74430607900001, 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:24.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.966 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 131.743955164, 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:24.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:24.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:24.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:24.972 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 131.737066844, 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:24.976 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.478785217999999, 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:24.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:24.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.468508802000002, 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:24.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.006557999000002, 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:24.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:24.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:24.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:24.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:24.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:24.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:24.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:24.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:24.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:24.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:24.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.994149733, 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.994117979000002, 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.990585482999997, 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.980523978000001, 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.766581070999997, 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.756569126999999, 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.483190264999998, 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.466521815999997, 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.280624033000002, 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.596865676, 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.574648677999999, 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.413427398, 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.27007884, 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2574538799999999, 2024-04-24T00:36:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:25.069 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:25.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:25.071 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:25.068+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:25.172 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:25.167+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:25.173 INFO:teuthology.orchestra.run.smithi064.stdout:4481 punch from 0x7641d87 to 0x764d64d, (0xb8c6 bytes) 2024-04-24T00:36:25.180 INFO:teuthology.orchestra.run.smithi064.stdout:4482 punch from 0x3b4d253 to 0x3b5bd06, (0xeab3 bytes) 2024-04-24T00:36:25.180 INFO:teuthology.orchestra.run.smithi064.stdout:4483 read 0x81f5dd thru 0x8242fe (0x4d22 bytes) 2024-04-24T00:36:25.181 INFO:teuthology.orchestra.run.smithi064.stdout:4485 read 0x3d8e10e thru 0x3d97913 (0x9806 bytes) 2024-04-24T00:36:25.182 INFO:teuthology.orchestra.run.smithi064.stdout:4486 read 0x3b55af6 thru 0x3b5e671 (0x8b7c bytes) 2024-04-24T00:36:25.182 INFO:teuthology.orchestra.run.smithi064.stdout:4490 punch from 0x39c6598 to 0x39c9595, (0x2ffd bytes) 2024-04-24T00:36:25.182 INFO:teuthology.orchestra.run.smithi064.stdout:4495 punch from 0x830546c to 0x830ad0d, (0x58a1 bytes) 2024-04-24T00:36:25.195 INFO:teuthology.orchestra.run.smithi064.stdout:4496 writesame 0xeaa89eb thru 0xeaa9eb2 data_size 0x14(0x14c8 bytes) 2024-04-24T00:36:25.195 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:25.195 INFO:teuthology.orchestra.run.smithi064.stdout:4498 write 0xe6cf7ab thru 0xe6ddd2d (0xe583 bytes) 2024-04-24T00:36:25.210 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:25.210 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:110585 3.13 3:c838fadb:::rbd_data.115e1702e7a8.00000000000001c8:head [set-alloc-hint object_size 524288 write_size 524288,write 491520~32768 in=32768b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:25.203967+0000", 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0057704139999999998, 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058480859999999997, 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 110585 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.203967+0000", 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.203967+0000", 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.203970+0000", 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3139999999999999e-06 2024-04-24T00:36:25.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.204018+0000", 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7905999999999998e-05 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.204121+0000", 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102736 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.204142+0000", 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0930000000000001e-05 2024-04-24T00:36:25.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.204287+0000", 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000145659 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.204585+0000", 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029784800000000003 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.204851+0000", 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026633400000000002 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:25.209394+0000", 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045426859999999998 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:36:25.215 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:25.216 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:25.222 INFO:teuthology.orchestra.run.smithi064.stdout:4500 trunc from 0xeaa9eb3 to 0x6c257e8 2024-04-24T00:36:25.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:25.265 INFO:teuthology.orchestra.run.smithi064.stdout:4501 write 0x41bdbbc thru 0x41c2195 (0x45da bytes) 2024-04-24T00:36:25.271 INFO:teuthology.orchestra.run.smithi064.stdout:4503 writesame 0xef332f thru 0xefa544 data_size 0x11(0x7216 bytes) 2024-04-24T00:36:25.273 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:25.268+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:25.297 INFO:teuthology.orchestra.run.smithi064.stdout:4504 write 0xdc1baa2 thru 0xdc23e85 (0x83e4 bytes) 2024-04-24T00:36:25.303 INFO:teuthology.orchestra.run.smithi064.stdout:4506 writesame 0x2157ad9 thru 0x215aa64 data_size 0x22(0x2f8c bytes) 2024-04-24T00:36:25.306 INFO:teuthology.orchestra.run.smithi064.stdout:4507 write 0xab8e381 thru 0xab9613f (0x7dbf bytes) 2024-04-24T00:36:25.316 INFO:teuthology.orchestra.run.smithi064.stdout:4509 punch from 0x3156322 to 0x3162ccc, (0xc9aa bytes) 2024-04-24T00:36:25.316 INFO:teuthology.orchestra.run.smithi064.stdout:4511 read 0xafbdd86 thru 0xafc6347 (0x85c2 bytes) 2024-04-24T00:36:25.317 INFO:teuthology.orchestra.run.smithi064.stdout:4512 punch from 0x7afc37 to 0x7b6f45, (0x730e bytes) 2024-04-24T00:36:25.317 INFO:teuthology.orchestra.run.smithi064.stdout:4513 read 0x3c9aeab thru 0x3ca93fa (0xe550 bytes) 2024-04-24T00:36:25.317 INFO:teuthology.orchestra.run.smithi064.stdout:4516 read 0xd7e0d7d thru 0xd7edd20 (0xcfa4 bytes) 2024-04-24T00:36:25.317 INFO:teuthology.orchestra.run.smithi064.stdout:4517 punch from 0x368a0ff to 0x36901da, (0x60db bytes) 2024-04-24T00:36:25.317 INFO:teuthology.orchestra.run.smithi064.stdout:4518 read 0xda420b4 thru 0xda50c7e (0xebcb bytes) 2024-04-24T00:36:25.323 INFO:teuthology.orchestra.run.smithi064.stdout:4519 trunc from 0xdc23e86 to 0xa362ce8 2024-04-24T00:36:25.339 INFO:teuthology.orchestra.run.smithi064.stdout:4520 write 0x7cb0c32 thru 0x7cb14db (0x8aa bytes) 2024-04-24T00:36:25.364 INFO:teuthology.orchestra.run.smithi064.stdout:4521 write 0xd34b631 thru 0xd34d40f (0x1ddf bytes) 2024-04-24T00:36:25.369 INFO:teuthology.orchestra.run.smithi064.stdout:4522 clone 30 order 19 su 32768 sc 12 2024-04-24T00:36:25.373 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:25.371+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:25.401 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:25.401 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:25.401 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:25.401 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:25.401 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:25.416 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:25.475 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:25.472+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:25.576 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:25.572+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:25.585 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:25.585 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:25.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:25.585 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:25.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.585 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:25.585 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:25.586 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.36611199000001, 2024-04-24T00:36:25.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:25.586 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.586 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.586 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:25.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:25.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 117.944326296, 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:25.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:25.597 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.802198730000001, 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:25.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.664426947999999, 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:25.604 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:25.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:25.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.610 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.626944629999997, 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.597615668000003, 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.617 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:36:25.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:36:25.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:36:25.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:36:25.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:36:25.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.400353764999998, 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.624 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:25.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:25.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:25.630 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.964598766000002, 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.631 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:25.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:25.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:25.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:25.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.954740729999997, 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.638 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:25.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.934391122000001, 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.644 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:25.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:25.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:25.650 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.650 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.650 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.900345786999999, 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:25.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:25.655 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.898035950999997, 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:25.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:25.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:25.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.661 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.889467152000002, 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:25.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.668 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.950318058999997, 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:25.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.675 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.060928062999999, 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:25.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:25.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:25.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:25.681 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.017452265999999, 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:25.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:25.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:25.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.043478964, 2024-04-24T00:36:25.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:25.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:25.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:25.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.692 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.024337061000001, 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:25.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:25.696 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.0650495080000004, 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:25.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:25.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:25.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:25.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 6.0310470860000001, 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:25.702 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:25.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:25.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:25.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:25.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:25.707 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.707 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.707 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.707 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:25.707 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:25.707 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:25.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:25.709 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:25.703+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:25.809 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:25.808+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:25.852 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:36:25.889 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:25.889 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:25.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:25.949 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:25.947+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:26.010 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:26.007+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:26.026 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:26.027 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:26.027 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:26.027 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:26.027 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:26.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:26.112 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:26.108+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:26.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:26.206 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.494721126, 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.494623201, 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:36:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:36:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:36:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 105.135243669, 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:36:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:36:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:36:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.312259682000004, 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:36:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:36:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:36:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.724499542999993, 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:36:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:36:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:36:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:36:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:36:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:36:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.699230936999996, 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:36:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.846027217, 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:36:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:36:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:36:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.813422102999994, 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:36:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.408847409000003, 2024-04-24T00:36:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:36:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.351007179000007, 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:36:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89931 3.1f 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head [write 0~4577672 [fadvise_sequential+fadvise_nocache],zero 4718592~262144,zero 6291456~179747,write 6471203~13759 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.511474575999998, 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068626284999999995, 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89931 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691618+0000", 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2949999999999998e-06 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704054+0000", 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012436665 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704090+0000", 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5778999999999997e-05 2024-04-24T00:36:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704107+0000", 2024-04-24T00:36:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7122999999999999e-05 2024-04-24T00:36:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704255+0000", 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014735099999999999 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704588+0000", 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033309499999999999 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.705598+0000", 2024-04-24T00:36:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001010597 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.758968+0000", 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053369708000000002 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760090+0000", 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011214840000000001 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760173+0000", 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3147999999999997e-05 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760241+0000", 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8040000000000006e-05 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.50944483, 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:36:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:36:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90259 3.2 3:4377c456:::rbd_data.115e56aa02fc.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.672961665000003, 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064637350999999996, 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90259 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530129+0000", 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530137+0000", 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530152+0000", 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4612e-05 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530160+0000", 2024-04-24T00:36:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-06 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530341+0000", 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018113600000000001 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530535+0000", 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193606 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.534329+0000", 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037947839999999998 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594447+0000", 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060117469999999999 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594580+0000", 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013329699999999999 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594679+0000", 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9085000000000004e-05 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594765+0000", 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5533999999999998e-05 2024-04-24T00:36:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.530697279000002, 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:36:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:36:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.501055225999998, 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:36:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:36:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.354539814999999, 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:36:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:36:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.565767256999999, 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.542623844, 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:36:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.6693150960000001, 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:36:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:36:26.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:36:26.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:26.325 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.6166328190000003, 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:26.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:26.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:26.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:26.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:36:26.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:36:26.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:26.332 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:26.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:26.335 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:26.331+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:26.352 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:26.352 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:26.370 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:26.370 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:26.388 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:26.388 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:26.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:26.406 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:26.425 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:26.435 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:26.432+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:26.443 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:26.463 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:26.485 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:26.485 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:26.515 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:26.515 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:26.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:26.545 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:26.544+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:26.637 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:26.635+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:26.649 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:26.649 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:26.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:26.749 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:26.747+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:26.828 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:26.828 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:26.838 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:26.836+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:26.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:26.940 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:26.935+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:27.009 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:27.009 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:27.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:27.066 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:27.063+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:27.142 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:27.140+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:27.188 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:27.188 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:27.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:27.242 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:27.240+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:27.319 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:27.320 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:27.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:27.343 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:27.339+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:27.444 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:27.440+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:27.482 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:27.482 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:27.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:27.590 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:27.588+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:27.646 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:27.643+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:27.673 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:27.673 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:27.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:27.747 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:27.743+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:27.848 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:27.844+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:27.873 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5, 0] out_osds: [7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:27.874 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:27.874 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-24T00:36:27.874 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:36:27.874 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:36:27.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-04-24T00:36:27.879 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:27.879 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:27.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:36:27.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:27.985 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:27.984+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:28.007 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:28.007 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T00:36:28.007 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:28.050 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:28.048+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:28.068 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:28.068 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:28.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:28.170 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:28.168+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:28.252 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:28.248+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:28.254 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:28.254 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:28.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:28.353 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:28.351+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:28.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:28.454 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:28.451+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:28.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:28.555 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:28.552+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:28.575 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:28.575 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:28.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:28.643 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:28.643 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:28.656 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:28.651+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:28.660 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:28.677 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:28.694 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:28.695 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:28.716 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:28.716 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:28.737 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:28.738 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:28.749 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:28.749 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:28.749 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:28.749 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:28.749 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:28.757 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:28.751+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:28.759 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:28.760 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:28.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:28.780 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:28.781 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:28.858 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:28.856+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:28.860 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:36:28.935 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.776555723999998, 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.936 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:28.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:28.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.940 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.751654684999998, 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:28.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.734910526999997, 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:28.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:28.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.644761533999997, 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:28.949 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:28.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:28.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.237604855000001, 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:28.954 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:28.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.227218157999999, 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:28.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.218274868999998, 2024-04-24T00:36:28.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:28.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105170 3.16 e152/112 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head v 152'511, mlcod=150'509)", 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.243581205, 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022303394000000001, 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689196+0000", 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7399999999999991e-07 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692342+0000", 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031461710000000001 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692378+0000", 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5707999999999998e-05 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692396+0000", 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8110000000000001e-05 2024-04-24T00:36:28.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692483+0000", 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6565999999999998e-05 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692513+0000", 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0151999999999999e-05 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711444+0000", 2024-04-24T00:36:28.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018930827000000001 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711499+0000", 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4886000000000003e-05 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105172 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [write 0~917504 [fadvise_sequential+fadvise_nocache],zero 917504~393216,write 1310720~131072 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.242941951000001, 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024973104999999999, 2024-04-24T00:36:28.973 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105172 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:28.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689837+0000", 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-06 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693186+0000", 2024-04-24T00:36:28.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033495899999999999 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693214+0000", 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7616000000000001e-05 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693229+0000", 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5424999999999999e-05 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693350+0000", 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120711 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693534+0000", 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018363200000000001 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693878+0000", 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000343958 2024-04-24T00:36:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.712994+0000", 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019116371 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714688+0000", 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016935539999999999 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714754+0000", 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6297000000000006e-05 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714807+0000", 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3522e-05 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.979 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105169 3.1f e152/59 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head v 152'697, mlcod=152'693)", 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.24026252, 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041348306000000001, 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692515+0000", 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699647+0000", 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071313169999999999 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699676+0000", 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9876999999999999e-05 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699693+0000", 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6973000000000002e-05 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699779+0000", 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5169999999999999e-05 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699805+0000", 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6120000000000001e-05 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733817+0000", 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034012112999999997 2024-04-24T00:36:28.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733862+0000", 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5252e-05 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.6920780149999999, 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:36:28.984 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:36:28.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:36:28.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.6623105339999995, 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:36:28.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:36:28.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:28.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106874 3.18 3:1ecbe779:::rbd_header.115ec2448e7a:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.816089+0000", 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1166875279999999, 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012895386999999999, 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:28.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106874 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816089+0000", 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816089+0000", 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816092+0000", 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.05e-06 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816107+0000", 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4983e-05 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816149+0000", 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1712000000000003e-05 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816174+0000", 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5612000000000001e-05 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816414+0000", 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023948599999999999 2024-04-24T00:36:28.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.816426+0000", 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2244e-05 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.825743+0000", 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093174759999999999 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.825895+0000", 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015211599999999999 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.826543+0000", 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064714000000000004 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.827851+0000", 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013083159999999999 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:28.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.828704+0000", 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085295399999999995 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.828877+0000", 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017332099999999999 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.828984+0000", 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106977 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109747 3.1f 3:fa7fa4bf:::rbd_data.115edc839ccf.000000000000006c:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1599821070000003, 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016743020000000001, 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109747 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772795+0000", 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772799+0000", 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.391e-06 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772811+0000", 2024-04-24T00:36:29.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1693e-05 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772821+0000", 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.004e-05 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777269+0000", 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044474149999999997 2024-04-24T00:36:29.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777464+0000", 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195353 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.789364+0000", 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011900617 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.789537+0000", 2024-04-24T00:36:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017274900000000001 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109751 3.1f 3:faff7380:::rbd_data.115edc839ccf.0000000000000070:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1599413619999996, 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017461602, 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:29.006 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109751 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:29.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772836+0000", 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772839+0000", 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.817e-06 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772851+0000", 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1378e-05 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772860+0000", 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6750000000000008e-06 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777515+0000", 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046550200000000002 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777744+0000", 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022927399999999999 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.790180+0000", 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012436107 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.790297+0000", 2024-04-24T00:36:29.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011663 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109761 3.12 3:4d42c299:::rbd_data.115edc839ccf.000000000000007a:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1599049209999999, 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015851832999999999, 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109761 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772872+0000", 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7100000000000001e-07 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772876+0000", 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8139999999999998e-06 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772888+0000", 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1833e-05 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772897+0000", 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775136+0000", 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022391020000000002 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775322+0000", 2024-04-24T00:36:29.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018560199999999999 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788599+0000", 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013277512999999999 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788723+0000", 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012385900000000001 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109770 3.18 3:1c37082e:::rbd_data.115edc839ccf.0000000000000083:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1598275090000003, 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020896860999999999, 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109770 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5899999999999996e-07 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772953+0000", 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9199999999999997e-06 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772964+0000", 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0964999999999999e-05 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772973+0000", 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9679999999999995e-06 2024-04-24T00:36:29.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.781771+0000", 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087973119999999998 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.781948+0000", 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000177398 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.793712+0000", 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011763852999999999 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.793846+0000", 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133786 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109775 3.18 3:1bef16aa:::rbd_data.115edc839ccf.0000000000000088:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1597906189999998, 2024-04-24T00:36:29.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021323353999999999, 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109775 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6400000000000002e-07 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772990+0000", 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.647e-06 2024-04-24T00:36:29.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773001+0000", 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1399999999999999e-05 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773011+0000", 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.782127+0000", 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091162650000000001 2024-04-24T00:36:29.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.782286+0000", 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015908 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.794246+0000", 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011959722000000001 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.794309+0000", 2024-04-24T00:36:29.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3305999999999993e-05 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109779 3.12 3:48246b4e:::rbd_data.115edc839ccf.000000000000008c:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1597550329999997, 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015522284000000001, 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:29.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109779 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:29.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773022+0000", 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.37e-07 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773025+0000", 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773036+0000", 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0635000000000001e-05 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773045+0000", 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6819999999999992e-06 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775372+0000", 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023270669999999999 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775551+0000", 2024-04-24T00:36:29.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017971 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788416+0000", 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012864123999999999 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788544+0000", 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012804 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109781 3.12 3:4dd90635:::rbd_data.115edc839ccf.000000000000008e:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.773091+0000", 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1596856779999998, 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015277236, 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109781 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773091+0000", 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773091+0000", 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773091+0000", 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6700000000000003e-07 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773095+0000", 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4869999999999998e-06 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773114+0000", 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8833000000000001e-05 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773123+0000", 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1500000000000005e-06 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775601+0000", 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024784579999999998 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775777+0000", 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017562200000000001 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788241+0000", 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012463805999999999 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788368+0000", 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731299999999999 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.034 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:29.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:29.036 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:29.032+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:29.136 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:29.132+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:29.206 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:29.206 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:29.206 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:29.206 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:29.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:29.237 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:29.235+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:29.338 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:29.336+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:29.386 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:29.387 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:29.387 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:29.387 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:29.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:29.438 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:29.435+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:29.539 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:29.536+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.356249389, 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.35589847400001, 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.34901015400001, 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.090728528, 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.080452112000003, 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.618501309000003, 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.606093043000001, 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.606061289000003, 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.602528792999998, 2024-04-24T00:36:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.592467288000002, 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.378524380999998, 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.368512437, 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.095133574999998, 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.078465125999998, 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.892567343000003, 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.208808986000001, 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.186591988, 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.025370708000001, 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.882022149999999, 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:36:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.869397190000001, 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:29.672 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:29.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:29.674 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:29.671+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:29.775 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:29.771+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:29.804 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:29.805 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:29.805 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:29.805 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:29.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:29.876 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:29.872+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:29.977 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:29.972+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:30.044 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:30.044 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:30.044 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:30.044 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:30.045 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:30.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:30.113 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:30.111+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:30.179 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:30.176+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 137.05027175199999, 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.62848605799999, 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.486358492000001, 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:30.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:30.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:30.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.348586709999999, 2024-04-24T00:36:30.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:30.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:30.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:30.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:30.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.311104391999997, 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.291 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:30.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.281775430000003, 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.297 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:36:30.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:36:30.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.084513526999999, 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.303 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.648758528000002, 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.309 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:30.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:30.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:30.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:30.314 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.638900491999998, 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:30.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:30.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.618550884000001, 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.320 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:30.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.584505548999999, 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:30.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.582195712999997, 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.335 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.573626914000002, 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:30.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:30.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.634477820999997, 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.341 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:30.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:30.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:30.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:30.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:30.347 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.745087824999999, 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:30.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:30.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:30.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.701612028, 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.353 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:30.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.727638725999999, 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.357 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:30.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:30.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:30.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:30.362 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.708496823000001, 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.74920927, 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.366 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.371 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.715206847999999, 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:30.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.375 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.376 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.376 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:30.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:30.378 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:30.375+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:30.478 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:30.475+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:30.544 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:30.544 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:30.544 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:30.544 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:30.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:30.601 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:30.599+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:30.681 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:30.675+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:30.723 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:30.723 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:30.723 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:30.723 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:30.723 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:30.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:30.781 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:30.779+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:30.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:30.883 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:30.880+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:30.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:36:30.915 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:30.915 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:30.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:30.915 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:30.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.20253591399999, 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:36:30.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:36:30.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.920 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.202437989, 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:36:30.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.926 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.843058457, 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.927 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:36:30.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:36:30.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:36:30.932 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.020074469999997, 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:36:30.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:36:30.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:36:30.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.937 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.432314331000001, 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.938 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:36:30.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:36:30.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:36:30.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.407045725000003, 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.943 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.553842005000007, 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.947 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:36:30.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:36:30.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:36:30.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:36:30.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:36:30.953 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.521236891000001, 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.954 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:36:30.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:30.958 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.116662196999997, 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:36:30.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:36:30.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.058821967, 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:36:30.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.967 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89931 3.1f 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head [write 0~4577672 [fadvise_sequential+fadvise_nocache],zero 4718592~262144,zero 6291456~179747,write 6471203~13759 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.219289363999998, 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068626284999999995, 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89931 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.968 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691618+0000", 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2949999999999998e-06 2024-04-24T00:36:30.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704054+0000", 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012436665 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704090+0000", 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5778999999999997e-05 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704107+0000", 2024-04-24T00:36:30.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7122999999999999e-05 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704255+0000", 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014735099999999999 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704588+0000", 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033309499999999999 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.705598+0000", 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001010597 2024-04-24T00:36:30.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.758968+0000", 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053369708000000002 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760090+0000", 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011214840000000001 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760173+0000", 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3147999999999997e-05 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760241+0000", 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8040000000000006e-05 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.217259618, 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:36:30.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90259 3.2 3:4377c456:::rbd_data.115e56aa02fc.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:30.977 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:30.978 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.380776453000003, 2024-04-24T00:36:30.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064637350999999996, 2024-04-24T00:36:30.978 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.978 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.978 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.978 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.978 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.978 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90259 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530129+0000", 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530137+0000", 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530152+0000", 2024-04-24T00:36:30.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4612e-05 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530160+0000", 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-06 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530341+0000", 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018113600000000001 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530535+0000", 2024-04-24T00:36:30.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193606 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.534329+0000", 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037947839999999998 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594447+0000", 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060117469999999999 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594580+0000", 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013329699999999999 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594679+0000", 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9085000000000004e-05 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594765+0000", 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5533999999999998e-05 2024-04-24T00:36:30.983 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.238512066999998, 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.984 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:36:30.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:36:30.989 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.208870013999999, 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.992 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.062354602999999, 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:36:30.993 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:36:30.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.273582045000001, 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:31.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:31.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 20.250438632000002, 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:31.005 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.377129884, 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:31.009 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:31.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:36:31.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:36:31.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 10.324447607, 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:31.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:31.018 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:31.019 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:31.020 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:31.020 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:31.016+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:31.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:31.030 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:31.030 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:31.061 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:31.061 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:31.078 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:31.079 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:31.100 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:31.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:31.111 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-04-24T00:36:31.120 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:31.115+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:31.128 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:31.128 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:31.149 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:31.149 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:31.149 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:31.149 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:31.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:31.164 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:31.201 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:31.201 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:31.253 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:31.251+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:31.322 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:31.320+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:31.341 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:31.342 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:31.356 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:31.441 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:31.439+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:31.481 INFO:teuthology.orchestra.run.smithi064.stdout:4525 write 0x64b9692 thru 0x64bccb8 (0x3627 bytes) 2024-04-24T00:36:31.494 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:31.494 INFO:teuthology.orchestra.run.smithi064.stdout:4528 read 0x77465de thru 0x775079a (0xa1bd bytes) 2024-04-24T00:36:31.494 INFO:teuthology.orchestra.run.smithi064.stdout:4529 write 0xa8d1083 thru 0xa8dff16 (0xee94 bytes) 2024-04-24T00:36:31.506 INFO:teuthology.orchestra.run.smithi064.stdout:4532 write 0x113d139 thru 0x1147dbe (0xac86 bytes) 2024-04-24T00:36:31.514 INFO:teuthology.orchestra.run.smithi064.stdout:4533 write 0x5931090 thru 0x5933e07 (0x2d78 bytes) 2024-04-24T00:36:31.520 INFO:teuthology.orchestra.run.smithi064.stdout:4534 punch from 0xae80895 to 0xae83989, (0x30f4 bytes) 2024-04-24T00:36:31.520 INFO:teuthology.orchestra.run.smithi064.stdout:4535 punch from 0x96ff901 to 0x9705361, (0x5a60 bytes) 2024-04-24T00:36:31.521 INFO:teuthology.orchestra.run.smithi064.stdout:4536 writesame 0x2ebf5c thru 0x2f62aa data_size 0x1(0xa34f bytes) 2024-04-24T00:36:31.523 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:31.519+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:31.534 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:31.534 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:31.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:31.611 INFO:teuthology.orchestra.run.smithi064.stdout:4537 trunc from 0xd34d410 to 0xa0deed5 2024-04-24T00:36:31.637 INFO:teuthology.orchestra.run.smithi064.stdout:4539 write 0x76fc5af thru 0x77047bd (0x820f bytes) 2024-04-24T00:36:31.645 INFO:teuthology.orchestra.run.smithi064.stdout:4540 write 0x264b644 thru 0x264bf9a (0x957 bytes) 2024-04-24T00:36:31.649 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:31.647+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:31.650 INFO:teuthology.orchestra.run.smithi064.stdout:4541 writesame 0x8a8be77 thru 0x8a937f3 data_size 0x7(0x797d bytes) 2024-04-24T00:36:31.666 INFO:teuthology.orchestra.run.smithi064.stdout:4544 trunc from 0xa0deed5 to 0xc0fdf0b 2024-04-24T00:36:31.682 INFO:teuthology.orchestra.run.smithi064.stdout:4545 read 0x9c7d4db thru 0x9c821da (0x4d00 bytes) 2024-04-24T00:36:31.682 INFO:teuthology.orchestra.run.smithi064.stdout:4547 write 0xb1334fa thru 0xb139e8b (0x6992 bytes) 2024-04-24T00:36:31.692 INFO:teuthology.orchestra.run.smithi064.stdout:4548 write 0x2646990 thru 0x2653654 (0xccc5 bytes) 2024-04-24T00:36:31.703 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:31.703 INFO:teuthology.orchestra.run.smithi064.stdout:4551 read 0x1f36862 thru 0x1f3c111 (0x58b0 bytes) 2024-04-24T00:36:31.704 INFO:teuthology.orchestra.run.smithi064.stdout:4552 clone 31 order 23 su 524288 sc 15 2024-04-24T00:36:31.725 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:31.720+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:31.730 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:31.730 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:31.744 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:31.829 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:31.827+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:31.895 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:36:31.910 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:31.910 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:31.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:31.927 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:31.923+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:32.027 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:32.024+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:32.095 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:32.095 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:32.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:32.153 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:32.152+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:32.228 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:32.223+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:32.244 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:32.244 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:32.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:32.346 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:32.344+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:32.428 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:32.428 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:32.430 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:32.428+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:32.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:32.531 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:32.528+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:32.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:32.632 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:32.627+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:32.733 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:32.728+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:32.799 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:32.799 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:32.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:32.834 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:32.831+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:32.935 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:32.932+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:33.018 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:33.018 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:33.023 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5, 0] out_osds: [7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:33.023 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:33.023 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T00:36:33.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T00:36:33.036 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:33.031+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:33.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:33.141 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:33.135+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:33.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:33.270 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:33.268+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:33.293 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:33.293 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:33.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:33.366 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:33.364+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:33.440 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:33.436+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:33.473 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:33.474 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:33.502 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:33.503 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:33.515 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:33.515 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:33.519 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:33.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:33.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:33.534 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:36:33.534 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:33.534 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 167, 2024-04-24T00:36:33.534 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:36:33.534 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-04-24T00:33:24.857711+0000", 2024-04-24T00:36:33.534 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-04-24T00:36:32.903523+0000", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-04-24T00:35:14.574643+0000", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-04-24T00:36:22.823938+0000", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5806080, 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 5, 2024-04-24T00:36:33.535 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 3, 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-04-24T00:36:33.536 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:39.734814+0000", 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-04-24T00:36:33.537 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "17", 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:33.538 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:33.539 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:33.540 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:33.541 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 8, 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 8, 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.25, 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-04-24T00:36:33.542 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:43.786311+0000", 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:33.543 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "20", 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:33.545 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:33.546 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.9900000095367432, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.9900000095367432, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.630000114440918, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.630000114440918, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:33.548 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "pool_client.0", 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:51.288702+0000", 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:33.549 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "167", 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 65, 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 167, 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:33.551 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:33.552 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:33.553 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.2400000095367432, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.2400000095367432, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-24T00:36:33.554 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1c80b3f4-99ee-4725-90a8-920116161bf5", 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:33.555 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 13, 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 74, 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 95, 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 163, 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 75, 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6817", 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6818", 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6819", 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6822", 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:33.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6823", 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6820", 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6821", 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6817/218259947", 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6819/218259947", 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6823/218259947", 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6821/218259947", 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:33.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "20e1331a-f8a2-48fd-b2b8-b7a92f21d78e", 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.264190673828125, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 159, 2024-04-24T00:36:33.561 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6824", 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6825", 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6826", 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6827", 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:33.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6830", 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6831", 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:33.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6828", 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6829", 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:33.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6825/637720411", 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6827/637720411", 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6831/637720411", 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6829/637720411", 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.566 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a51557a0-9634-4a20-82b0-b54cdd36b166", 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 162, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.567 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/820024667", 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6811/820024667", 2024-04-24T00:36:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6815/820024667", 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6813/820024667", 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "aafffa79-f532-4441-b1d6-f6441c5eebf8", 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.79296875, 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.3392791748046875, 2024-04-24T00:36:33.572 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 161, 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/1974534323", 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6803/1974534323", 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1974534323", 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1974534323", 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "78f05556-598e-4be2-b682-e3f58863eb82", 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.9623870849609375, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 162, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6816", 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6818", 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.579 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6820", 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6821", 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.580 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6826", 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6828", 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:33.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6823", 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6824", 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6818/1642543604", 2024-04-24T00:36:33.582 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6821/1642543604", 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6828/1642543604", 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6824/1642543604", 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5", 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:33.583 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 142, 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6808", 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6809", 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6810", 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:33.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6811", 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6814", 2024-04-24T00:36:33.586 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6815", 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6812", 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6813", 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6809/172726194", 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6811/172726194", 2024-04-24T00:36:33.588 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6815/172726194", 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6813/172726194", 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "6dad12bf-94de-482a-a402-49e67d5e5dc7", 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 160, 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6800", 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6801", 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6802", 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6803", 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6806", 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6807", 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6804", 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6805", 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6801/2195746987", 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6803/2195746987", 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2195746987", 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6805/2195746987", 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:33.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e6734cfe-3c24-4ddd-8ef0-5ed450efe772", 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 62, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:33.595 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6817", 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6819", 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.596 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6822", 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6825", 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.597 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6830", 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6831", 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6827", 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6829", 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.599 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6819/954791674", 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6825/954791674", 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6831/954791674", 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6829/954791674", 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:33.600 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.509233+0000", 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 75 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:33.601 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.465696+0000", 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.405177+0000", 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.602 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.269202+0000", 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:33.603 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.359180+0000", 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.351834+0000", 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:33.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.490127+0000", 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:36:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.484842+0000", 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "2.3", 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:33.606 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: 3 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.16", 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: 2 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.1b", 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:33.607 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-04-24T00:36:33.608 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [ 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 61, 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.609 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 63, 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-04-24T00:36:33.610 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-04-24T00:36:33.611 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-04-24T00:36:33.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.611 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:33.612 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:33.612 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:33.613 INFO:tasks.thrashosds.thrasher:j is {'epoch': 167, 'fsid': '6a70d87d-4453-47be-af2a-5590f9851254', 'created': '2024-04-24T00:33:24.857711+0000', 'modified': '2024-04-24T00:36:32.903523+0000', 'last_up_change': '2024-04-24T00:35:14.574643+0000', 'last_in_change': '2024-04-24T00:36:22.823938+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T00:33:39.734814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T00:33:43.786311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-24T00:33:51.288702+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 65, 'snap_epoch': 167, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '1c80b3f4-99ee-4725-90a8-920116161bf5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 74, 'up_from': 95, 'up_thru': 163, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 218259947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 218259947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 218259947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 218259947}]}, 'public_addr': '172.21.15.64:6817/218259947', 'cluster_addr': '172.21.15.64:6819/218259947', 'heartbeat_back_addr': '172.21.15.64:6823/218259947', 'heartbeat_front_addr': '172.21.15.64:6821/218259947', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'up': 1, 'in': 1, 'weight': 0.264190673828125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 159, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 637720411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 637720411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 637720411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 637720411}]}, 'public_addr': '172.21.15.64:6825/637720411', 'cluster_addr': '172.21.15.64:6827/637720411', 'heartbeat_back_addr': '172.21.15.64:6831/637720411', 'heartbeat_front_addr': '172.21.15.64:6829/637720411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a51557a0-9634-4a20-82b0-b54cdd36b166', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 820024667}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 820024667}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 820024667}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 820024667}]}, 'public_addr': '172.21.15.64:6809/820024667', 'cluster_addr': '172.21.15.64:6811/820024667', 'heartbeat_back_addr': '172.21.15.64:6815/820024667', 'heartbeat_front_addr': '172.21.15.64:6813/820024667', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aafffa79-f532-4441-b1d6-f6441c5eebf8', 'up': 1, 'in': 1, 'weight': 0.79296875, 'primary_affinity': 0.3392791748046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 161, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1974534323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1974534323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1974534323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1974534323}]}, 'public_addr': '172.21.15.64:6801/1974534323', 'cluster_addr': '172.21.15.64:6803/1974534323', 'heartbeat_back_addr': '172.21.15.64:6807/1974534323', 'heartbeat_front_addr': '172.21.15.64:6805/1974534323', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '78f05556-598e-4be2-b682-e3f58863eb82', 'up': 1, 'in': 1, 'weight': 0.9623870849609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6816', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6818', 'nonce': 1642543604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6820', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6821', 'nonce': 1642543604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6826', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6828', 'nonce': 1642543604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6823', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6824', 'nonce': 1642543604}]}, 'public_addr': '172.21.15.167:6818/1642543604', 'cluster_addr': '172.21.15.167:6821/1642543604', 'heartbeat_back_addr': '172.21.15.167:6828/1642543604', 'heartbeat_front_addr': '172.21.15.167:6824/1642543604', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 142, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6808', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 172726194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6810', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 172726194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6814', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 172726194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6812', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 172726194}]}, 'public_addr': '172.21.15.167:6809/172726194', 'cluster_addr': '172.21.15.167:6811/172726194', 'heartbeat_back_addr': '172.21.15.167:6815/172726194', 'heartbeat_front_addr': '172.21.15.167:6813/172726194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6dad12bf-94de-482a-a402-49e67d5e5dc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6800', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2195746987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6802', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2195746987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6806', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2195746987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6804', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2195746987}]}, 'public_addr': '172.21.15.167:6801/2195746987', 'cluster_addr': '172.21.15.167:6803/2195746987', 'heartbeat_back_addr': '172.21.15.167:6807/2195746987', 'heartbeat_front_addr': '172.21.15.167:6805/2195746987', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e6734cfe-3c24-4ddd-8ef0-5ed450efe772', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6817', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6819', 'nonce': 954791674}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6822', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6825', 'nonce': 954791674}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6830', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6831', 'nonce': 954791674}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6827', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6829', 'nonce': 954791674}]}, 'public_addr': '172.21.15.167:6819/954791674', 'cluster_addr': '172.21.15.167:6825/954791674', 'heartbeat_back_addr': '172.21.15.167:6831/954791674', 'heartbeat_front_addr': '172.21.15.167:6829/954791674', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.509233+0000', 'dead_epoch': 75}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.465696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.405177+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.269202+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.359180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.351834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.490127+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.484842+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.3', 'osds': [7, 3]}, {'pgid': '3.16', 'osds': [3, 2]}, {'pgid': '3.1b', 'osds': [5, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 61, 'length': 1}, {'begin': 63, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T00:36:33.614 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.3 2024-04-24T00:36:33.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap 2.3 2024-04-24T00:36:33.614 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:33.614 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:33.615 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:33.611+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:33.713 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:33.712+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:33.720 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:33.720 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:33.720 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:33.720 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:33.721 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:33.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:33.786 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:33.787 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:33.807 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:33.807 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:33.814 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:33.812+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:33.828 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:33.828 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:33.890 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:33.890 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.730123788, 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:33.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.705222749000001, 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.688478590999999, 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:33.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:33.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:33.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:33.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.598329597999999, 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.906 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:33.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:33.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:33.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:33.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.191172919000003, 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:33.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:33.917 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.180786222000002, 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:33.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:33.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:33.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:33.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:33.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.171842933000001, 2024-04-24T00:36:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:33.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105170 3.16 e152/112 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head v 152'511, mlcod=150'509)", 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.197149269000001, 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022303394000000001, 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689196+0000", 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7399999999999991e-07 2024-04-24T00:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692342+0000", 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031461710000000001 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692378+0000", 2024-04-24T00:36:33.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5707999999999998e-05 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692396+0000", 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8110000000000001e-05 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692483+0000", 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6565999999999998e-05 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692513+0000", 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0151999999999999e-05 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711444+0000", 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018930827000000001 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711499+0000", 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4886000000000003e-05 2024-04-24T00:36:33.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105172 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [write 0~917504 [fadvise_sequential+fadvise_nocache],zero 917504~393216,write 1310720~131072 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:33.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:33.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.196510015000001, 2024-04-24T00:36:33.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024973104999999999, 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105172 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689837+0000", 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-06 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693186+0000", 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033495899999999999 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693214+0000", 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7616000000000001e-05 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693229+0000", 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5424999999999999e-05 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693350+0000", 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120711 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693534+0000", 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018363200000000001 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693878+0000", 2024-04-24T00:36:33.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000343958 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.712994+0000", 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019116371 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714688+0000", 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016935539999999999 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714754+0000", 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6297000000000006e-05 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714807+0000", 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3522e-05 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105169 3.1f e152/59 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head v 152'697, mlcod=152'693)", 2024-04-24T00:36:33.939 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.193830584000001, 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041348306000000001, 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:33.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692515+0000", 2024-04-24T00:36:33.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:36:33.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699647+0000", 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071313169999999999 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699676+0000", 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9876999999999999e-05 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699693+0000", 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6973000000000002e-05 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699779+0000", 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5169999999999999e-05 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699805+0000", 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6120000000000001e-05 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733817+0000", 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034012112999999997 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733862+0000", 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5252e-05 2024-04-24T00:36:33.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.645646079, 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:36:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:36:33.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.615878598, 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:33.952 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:33.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:33.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109747 3.1f 3:fa7fa4bf:::rbd_data.115edc839ccf.000000000000006c:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.113550171, 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016743020000000001, 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109747 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.957 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772795+0000", 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772799+0000", 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.391e-06 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772811+0000", 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1693e-05 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772821+0000", 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.004e-05 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777269+0000", 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044474149999999997 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777464+0000", 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195353 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.789364+0000", 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011900617 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.789537+0000", 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017274900000000001 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109751 3.1f 3:faff7380:::rbd_data.115edc839ccf.0000000000000070:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1135094260000002, 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017461602, 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109751 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772836+0000", 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772839+0000", 2024-04-24T00:36:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.817e-06 2024-04-24T00:36:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772851+0000", 2024-04-24T00:36:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1378e-05 2024-04-24T00:36:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772860+0000", 2024-04-24T00:36:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6750000000000008e-06 2024-04-24T00:36:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777515+0000", 2024-04-24T00:36:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046550200000000002 2024-04-24T00:36:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777744+0000", 2024-04-24T00:36:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022927399999999999 2024-04-24T00:36:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.790180+0000", 2024-04-24T00:36:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012436107 2024-04-24T00:36:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.790297+0000", 2024-04-24T00:36:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011663 2024-04-24T00:36:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109761 3.12 3:4d42c299:::rbd_data.115edc839ccf.000000000000007a:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1134729849999996, 2024-04-24T00:36:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015851832999999999, 2024-04-24T00:36:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109761 2024-04-24T00:36:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772872+0000", 2024-04-24T00:36:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7100000000000001e-07 2024-04-24T00:36:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772876+0000", 2024-04-24T00:36:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8139999999999998e-06 2024-04-24T00:36:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772888+0000", 2024-04-24T00:36:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1833e-05 2024-04-24T00:36:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772897+0000", 2024-04-24T00:36:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:36:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775136+0000", 2024-04-24T00:36:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022391020000000002 2024-04-24T00:36:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775322+0000", 2024-04-24T00:36:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018560199999999999 2024-04-24T00:36:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788599+0000", 2024-04-24T00:36:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013277512999999999 2024-04-24T00:36:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788723+0000", 2024-04-24T00:36:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012385900000000001 2024-04-24T00:36:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109770 3.18 3:1c37082e:::rbd_data.115edc839ccf.0000000000000083:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.113395573, 2024-04-24T00:36:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020896860999999999, 2024-04-24T00:36:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109770 2024-04-24T00:36:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5899999999999996e-07 2024-04-24T00:36:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772953+0000", 2024-04-24T00:36:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9199999999999997e-06 2024-04-24T00:36:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772964+0000", 2024-04-24T00:36:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0964999999999999e-05 2024-04-24T00:36:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772973+0000", 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9679999999999995e-06 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.781771+0000", 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087973119999999998 2024-04-24T00:36:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.781948+0000", 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000177398 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.793712+0000", 2024-04-24T00:36:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011763852999999999 2024-04-24T00:36:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.793846+0000", 2024-04-24T00:36:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133786 2024-04-24T00:36:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109775 3.18 3:1bef16aa:::rbd_data.115edc839ccf.0000000000000088:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1133586829999995, 2024-04-24T00:36:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021323353999999999, 2024-04-24T00:36:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109775 2024-04-24T00:36:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6400000000000002e-07 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772990+0000", 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.647e-06 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773001+0000", 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1399999999999999e-05 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773011+0000", 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.782127+0000", 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091162650000000001 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.782286+0000", 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015908 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.794246+0000", 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011959722000000001 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.794309+0000", 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3305999999999993e-05 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109779 3.12 3:48246b4e:::rbd_data.115edc839ccf.000000000000008c:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1133230970000003, 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015522284000000001, 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109779 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773022+0000", 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.37e-07 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773025+0000", 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773036+0000", 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0635000000000001e-05 2024-04-24T00:36:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773045+0000", 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6819999999999992e-06 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775372+0000", 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023270669999999999 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775551+0000", 2024-04-24T00:36:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017971 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788416+0000", 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012864123999999999 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788544+0000", 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012804 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109781 3.12 3:4dd90635:::rbd_data.115edc839ccf.000000000000008e:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.773091+0000", 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1132537419999995, 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015277236, 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109781 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773091+0000", 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773091+0000", 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773091+0000", 2024-04-24T00:36:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6700000000000003e-07 2024-04-24T00:36:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773095+0000", 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4869999999999998e-06 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773114+0000", 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8833000000000001e-05 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773123+0000", 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1500000000000005e-06 2024-04-24T00:36:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775601+0000", 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024784579999999998 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775777+0000", 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017562200000000001 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788241+0000", 2024-04-24T00:36:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012463805999999999 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788368+0000", 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731299999999999 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111494 3.4 3:2495b521:::rbd_data.115e397a9cd4.0000000000000015:head [write 0~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:29.953637+0000", 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9327073219999997, 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013109437999999999, 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111494 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.953637+0000", 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.953637+0000", 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.953640+0000", 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6010000000000002e-06 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.954255+0000", 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061504799999999998 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.954292+0000", 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7109e-05 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.954311+0000", 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-04-24T00:36:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.954424+0000", 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011285699999999999 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.954872+0000", 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044861899999999998 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.955116+0000", 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024419900000000001 2024-04-24T00:36:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.961800+0000", 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066838009999999996 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.966431+0000", 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046309360000000004 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.966676+0000", 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024487899999999998 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.966747+0000", 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0488999999999995e-05 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.013 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:34.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:36:34.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:34.016 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:34.012+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:34.089 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-04-24T00:36:34.138 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:34.136+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:34.194 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:34.194 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:34.194 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:34.194 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:34.211 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:34.216 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:34.211+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:34.317 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:34.312+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:34.367 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:34.367 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:34.367 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:34.367 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:34.367 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:34.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:34.422 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:34.420+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:34.518 INFO:teuthology.orchestra.run.smithi064.stdout:4553 read 0xbee4735 thru 0xbeee7cb (0xa097 bytes) 2024-04-24T00:36:34.520 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:34.515+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:34.527 INFO:teuthology.orchestra.run.smithi064.stdout:4556 trunc from 0xc0fdf0b to 0x4b3f37a 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 141.322428546, 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.541 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 141.32207763100001, 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 141.31518931100001, 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.553 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:34.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:34.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:34.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.056907684999999, 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:34.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:34.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.046631269000002, 2024-04-24T00:36:34.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:34.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:34.567 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.584680466000002, 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.568 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.5722722, 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.574 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:34.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:34.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.572240446000002, 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:34.578 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.568707949999997, 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.558646445000001, 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:34.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:34.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:34.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.344703537999997, 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:34.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:34.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:34.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:34.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.334691593999999, 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.061312731999998, 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.602 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:34.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.044644282999997, 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:34.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.611 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.858746500000002, 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.612 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:34.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:34.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.174988143, 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.618 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:34.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:34.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.152771145000003, 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:34.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.628 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.991549865, 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.629 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.635 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.848201307, 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:34.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:34.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:34.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:36:34.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.835576347, 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:34.642 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:34.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:36:34.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:34.646 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:34.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:34.649 INFO:teuthology.orchestra.run.smithi064.stdout:4557 write 0x741632 thru 0x750438 (0xee07 bytes) 2024-04-24T00:36:34.649 INFO:teuthology.orchestra.run.smithi064.stdout:4558 trunc from 0x4b3f37a to 0xd9e7b9d 2024-04-24T00:36:34.649 INFO:teuthology.orchestra.run.smithi064.stdout:4559 read 0x571f46d thru 0x5728e90 (0x9a24 bytes) 2024-04-24T00:36:34.649 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:34.649 INFO:teuthology.orchestra.run.smithi064.stdout:4562 read 0xcf94b7b thru 0xcf9f28b (0xa711 bytes) 2024-04-24T00:36:34.650 INFO:teuthology.orchestra.run.smithi064.stdout:4563 trunc from 0xd9e7b9d to 0xc8cd983 2024-04-24T00:36:34.650 INFO:teuthology.orchestra.run.smithi064.stdout:4564 read 0xafd925e thru 0xafe40eb (0xae8e bytes) 2024-04-24T00:36:34.650 INFO:teuthology.orchestra.run.smithi064.stdout:4567 writesame 0xe275c4 thru 0xe34f82 data_size 0x11(0xd9bf bytes) 2024-04-24T00:36:34.650 INFO:teuthology.orchestra.run.smithi064.stdout:4569 write 0xb71de1a thru 0xb721f9a (0x4181 bytes) 2024-04-24T00:36:34.650 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:34.644+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:34.654 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:34.654 INFO:teuthology.orchestra.run.smithi064.stdout:4571 write 0xbf7cd98 thru 0xbf85537 (0x87a0 bytes) 2024-04-24T00:36:34.666 INFO:teuthology.orchestra.run.smithi064.stdout:4572 write 0x92b8050 thru 0x92bde7a (0x5e2b bytes) 2024-04-24T00:36:34.670 INFO:teuthology.orchestra.run.smithi064.stdout:4573 writesame 0xba71608 thru 0xba790fd data_size 0x1(0x7af6 bytes) 2024-04-24T00:36:34.748 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:34.743+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:34.784 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:34.784 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:34.784 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:34.792 INFO:teuthology.orchestra.run.smithi064.stdout:4574 punch from 0x369a819 to 0x36a696e, (0xc155 bytes) 2024-04-24T00:36:34.793 INFO:teuthology.orchestra.run.smithi064.stdout:4575 punch from 0x2ac096 to 0x2afc60, (0x3bca bytes) 2024-04-24T00:36:34.793 INFO:teuthology.orchestra.run.smithi064.stdout:4576 write 0x9122a0a thru 0x912946a (0x6a61 bytes) 2024-04-24T00:36:34.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:34.812 INFO:teuthology.orchestra.run.smithi064.stdout:4578 trunc from 0xc8cd983 to 0x8e06ec0 2024-04-24T00:36:34.828 INFO:teuthology.orchestra.run.smithi064.stdout:4579 read 0x3719ac0 thru 0x37233cd (0x990e bytes) 2024-04-24T00:36:34.830 INFO:teuthology.orchestra.run.smithi064.stdout:4580 write 0x8a1f1ae thru 0x8a23111 (0x3f64 bytes) 2024-04-24T00:36:34.839 INFO:teuthology.orchestra.run.smithi064.stdout:4582 trunc from 0x8e06ec0 to 0xa7a9651 2024-04-24T00:36:34.854 INFO:teuthology.orchestra.run.smithi064.stdout:4583 punch from 0x2e8468d to 0x2e8532a, (0xc9d bytes) 2024-04-24T00:36:34.854 INFO:teuthology.orchestra.run.smithi064.stdout:4584 read 0x77ded20 thru 0x77e725e (0x853f bytes) 2024-04-24T00:36:34.855 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:34.872 INFO:teuthology.orchestra.run.smithi064.stdout:4586 writesame 0xc32af99 thru 0xc32f372 data_size 0x2d(0x43da bytes) 2024-04-24T00:36:34.872 INFO:teuthology.orchestra.run.smithi064.stdout:4587 write 0xaea9009 thru 0xaeb8ceb (0xfce3 bytes) 2024-04-24T00:36:34.882 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:34.889 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:34.887+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:34.907 INFO:teuthology.orchestra.run.smithi064.stdout:4589 trunc from 0xc32f373 to 0x2c136b6 2024-04-24T00:36:34.933 INFO:teuthology.orchestra.run.smithi064.stderr:clear 2.3 pg_upmap mapping 2024-04-24T00:36:34.934 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:36:34.950 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:34.947+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:34.960 INFO:teuthology.orchestra.run.smithi064.stdout:4590 write 0x35f20c7 thru 0x35f34f7 (0x1431 bytes) 2024-04-24T00:36:34.974 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:34.974 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:34.974 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:34.974 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:34.974 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:34.988 INFO:teuthology.orchestra.run.smithi064.stdout:4591 writesame 0x92ca89c thru 0x92cf210 data_size 0x1(0x4975 bytes) 2024-04-24T00:36:34.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:35.001 INFO:teuthology.orchestra.run.smithi064.stdout:4592 trunc from 0x92cf211 to 0xea8720 2024-04-24T00:36:35.018 INFO:teuthology.orchestra.run.smithi064.stdout:4593 read 0x75f01b thru 0x763c24 (0x4c0a bytes) 2024-04-24T00:36:35.030 INFO:teuthology.orchestra.run.smithi064.stdout:4594 trunc from 0xea8720 to 0x9114673 2024-04-24T00:36:35.040 INFO:teuthology.orchestra.run.smithi064.stdout:4595 writesame 0x74916f8 thru 0x74a0ea5 data_size 0x31(0xf7ae bytes) 2024-04-24T00:36:35.055 INFO:teuthology.orchestra.run.smithi064.stdout:4596 write 0xb640944 thru 0xb64be0c (0xb4c9 bytes) 2024-04-24T00:36:35.079 INFO:teuthology.orchestra.run.smithi064.stdout:4597 trunc from 0xb64be0d to 0xb782ae 2024-04-24T00:36:35.081 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:35.079+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:35.126 INFO:teuthology.orchestra.run.smithi064.stdout:4598 writesame 0x1b815f0 thru 0x1b8b114 data_size 0x9(0x9b25 bytes) 2024-04-24T00:36:35.129 INFO:teuthology.orchestra.run.smithi064.stdout:4599 clone 32 order 20 su 131072 sc 15 2024-04-24T00:36:35.152 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:35.147+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:35.163 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:35.163 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:35.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.94425242899999, 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.164 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:35.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:35.169 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.52246673499999, 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.380339168999996, 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.175 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:35.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:35.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:35.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:35.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:35.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:35.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.242567387000001, 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.184 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:35.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:35.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:35.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:35.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:35.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:35.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.190 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.205085068999999, 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.191 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:35.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:35.196 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.175756106999998, 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:36:35.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:36:35.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:36:35.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.978494204, 2024-04-24T00:36:35.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:35.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:35.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.542739204999997, 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:35.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:35.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:35.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.532881168999999, 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.215 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:35.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.512531561000003, 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:35.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.478486226000001, 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:35.227 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:35.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.476176389999999, 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.232 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:35.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:35.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.467607590999997, 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.239 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:35.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:35.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:35.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.528458497999999, 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.245 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:35.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:35.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.639068502000001, 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:35.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:35.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:35.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.595592705000001, 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:35.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.621619403, 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.262 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:35.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:35.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.602477499999999, 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:35.268 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:35.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:35.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.643189947, 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:35.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.609187524999999, 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:35.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:35.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:35.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:35.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.282 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.282 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.282 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:35.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:35.284 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:35.280+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:35.384 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:35.380+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:35.452 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:35.452 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:35.453 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:35.453 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:35.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:35.485 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:35.484+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:35.586 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:35.584+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:35.590 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:35.590 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:35.590 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:35.590 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:35.590 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:35.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:35.687 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:35.683+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:35.731 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:35.731 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:35.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:35.731 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:35.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 128.01824459900001, 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:36:35.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 128.01814667400001, 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.737 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:36:35.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:36:35.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.658767142, 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.744 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:36:35.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:35.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:35.749 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 98.835783155000001, 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:36:35.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 91.248023016000005, 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:36:35.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:36:35.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:35.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 91.222754409999993, 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:35.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:36:35.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.765 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 77.369550689999997, 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.766 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:36:35.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 77.336945576000005, 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.775 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.932370882000001, 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.776 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:36:35.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:36:35.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:36:35.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.874530652000004, 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:35.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:36:35.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:36:35.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89931 3.1f 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head [write 0~4577672 [fadvise_sequential+fadvise_nocache],zero 4718592~262144,zero 6291456~179747,write 6471203~13759 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.034998049000002, 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068626284999999995, 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.786 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89931 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691618+0000", 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2949999999999998e-06 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704054+0000", 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012436665 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704090+0000", 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5778999999999997e-05 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704107+0000", 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7122999999999999e-05 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704255+0000", 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014735099999999999 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704588+0000", 2024-04-24T00:36:35.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033309499999999999 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.705598+0000", 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001010597 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.758968+0000", 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053369708000000002 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760090+0000", 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011214840000000001 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760173+0000", 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3147999999999997e-05 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760241+0000", 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8040000000000006e-05 2024-04-24T00:36:35.791 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.032968302999997, 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:35.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90259 3.2 3:4377c456:::rbd_data.115e56aa02fc.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:35.796 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.196485138, 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064637350999999996, 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90259 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530129+0000", 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530137+0000", 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530152+0000", 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4612e-05 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530160+0000", 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-06 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530341+0000", 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018113600000000001 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530535+0000", 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193606 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.534329+0000", 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037947839999999998 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594447+0000", 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060117469999999999 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594580+0000", 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013329699999999999 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594679+0000", 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9085000000000004e-05 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594765+0000", 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5533999999999998e-05 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 30.054220751999999, 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.802 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 30.024578698999999, 2024-04-24T00:36:35.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:36:35.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.878063288, 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:36:35.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:36:35.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:36:35.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:35.820 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.089290729999998, 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:36:35.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.066147316999999, 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:36:35.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:36:35.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.192838568999999, 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:36:35.831 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:36:35.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.140156292, 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:35.837 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:36:35.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:36:35.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:35.841 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:35.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:35.843 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:35.840+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:35.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:35.943 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:35.940+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:36.010 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:36.010 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:36.019 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:36.019 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:36.019 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:36.019 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:36.028 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:36.028 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:36.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:36.043 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:36.040+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:36.045 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:36.045 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:36.061 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:36.061 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:36.078 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:36.079 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:36.097 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:36.098 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:36.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:36.118 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:36.138 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:36.138 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:36.144 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:36.139+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:36.158 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:36.158 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:36.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:36.245 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:36.240+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:36.339 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:36.339 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:36.346 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:36.343+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:36.356 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:36.449 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:36.447+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:36.525 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:36.525 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:36.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:36.548 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:36.544+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:36.649 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:36.647+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:36.701 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:36.701 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:36.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:36.751 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:36.748+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:36.851 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:36.847+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:36.878 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:36.878 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:36.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:36.952 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:36.947+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:36.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:36:37.053 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:37.048+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:37.068 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:37.068 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:37.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:37.177 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:37.176+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:37.248 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:37.248 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:37.255 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:37.251+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:37.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:37.356 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:37.352+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:37.433 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:37.434 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:37.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:37.490 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:37.488+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:37.557 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:37.552+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:37.617 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:37.617 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:37.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:37.674 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:37.672+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:37.759 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:37.755+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:37.804 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:37.804 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:37.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:37.865 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:37.864+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:37.946 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:37.947 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:37.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:37.963 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:37.956+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:37.967 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:36:38.052 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-04-24T00:36:38.062 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:38.059+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:38.097 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:38.097 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:38.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:38.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:38.167 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:38.159+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:38.253 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:38.253 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:38.263 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:38.259+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:38.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:38.314 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:38.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:38.332 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:38.332 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:38.359 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:38.359 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:38.364 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:38.359+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:38.377 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:38.377 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:38.395 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:38.395 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:38.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:38.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:38.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84125 3.9 e111/21 3:96523fd8:::rbd_data.115e6f07caf6.000000000000000a:head v 111'391, mlcod=111'389)", 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.235293677000001, 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040935745000000003, 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.156221+0000", 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160958+0000", 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047361590000000002 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160976+0000", 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8660000000000001e-05 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.160985+0000", 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4940000000000003e-06 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161052+0000", 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7297000000000003e-05 2024-04-24T00:36:38.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.161072+0000", 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9707e-05 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197086+0000", 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036014675000000003 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197156+0000", 2024-04-24T00:36:38.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9949000000000004e-05 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84133 3.1f e111/59 3:fe9dc2b8:::rbd_data.115e6f07caf6.000000000000000e:head v 111'653, mlcod=108'651)", 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.210392638000002, 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051458776999999997, 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181122+0000", 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.181124+0000", 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.638e-06 2024-04-24T00:36:38.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187248+0000", 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061237660000000001 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187270+0000", 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2135999999999999e-05 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187283+0000", 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2532e-05 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187365+0000", 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1886999999999997e-05 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.187389+0000", 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4471e-05 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232524+0000", 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045134690999999998 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.232580+0000", 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6656000000000001e-05 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84135 3.0 e111/21 3:07474c31:::rbd_data.115e6f07caf6.0000000000000010:head v 111'278, mlcod=108'276)", 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.193648479999993, 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053072102000000003, 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197866+0000", 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.197867+0000", 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199992+0000", 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021250700000000002 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200341+0000", 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034926100000000002 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200371+0000", 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9720999999999999e-05 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200476+0000", 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105418 2024-04-24T00:36:38.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.200503+0000", 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6452e-05 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250899+0000", 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050396390999999999 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.250938+0000", 2024-04-24T00:36:38.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8636000000000001e-05 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:85157 3.1f e111/59 3:fc556f4a:::rbd_data.115e6f07caf6.0000000000000015:head v 111'654, mlcod=108'652)", 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.103499486999993, 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027352587000000001, 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288015+0000", 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.288016+0000", 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9500000000000001e-07 2024-04-24T00:36:38.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291766+0000", 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037501079999999998 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291790+0000", 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853e-05 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291800+0000", 2024-04-24T00:36:38.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0071999999999999e-05 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291867+0000", 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7664999999999997e-05 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.291888+0000", 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0661000000000001e-05 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315317+0000", 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023428805 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.315367+0000", 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0528000000000001e-05 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89934 3.18 3:1dc21e0a:::rbd_data.115e56aa02fc.0000000000000006:head [write 0~4194304 [fadvise_sequential+fadvise_nocache],zero 4194304~262144,zero 5505024~262144,zero 6291456~524288,zero 7077888~106390,write 7184278~59773 [fadvise_sequential+fadvise_nocache],zero 7864320~255181,write 8119501~17347 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:38.413 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.696342807999997, 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044598462999999998, 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89934 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695171+0000", 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.695174+0000", 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7369999999999998e-06 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698839+0000", 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003664844 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698891+0000", 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1746000000000002e-05 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.698905+0000", 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4603999999999999e-05 2024-04-24T00:36:38.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699023+0000", 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011806700000000001 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699237+0000", 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021346900000000001 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.699916+0000", 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067939599999999997 2024-04-24T00:36:38.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739194+0000", 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039277701999999998 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739680+0000", 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048629599999999999 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739729+0000", 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9008999999999998e-05 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.739770+0000", 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0593000000000003e-05 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89931 3.1f e125/59 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head v 125'667, mlcod=113'665)", 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.685956111000003, 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042682757000000002, 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705558+0000", 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.705559+0000", 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0900000000000001e-07 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718051+0000", 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012492191 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718075+0000", 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4334000000000001e-05 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718099+0000", 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718193+0000", 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4630999999999996e-05 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.718227+0000", 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748185+0000", 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029957946999999999 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.748241+0000", 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6249000000000003e-05 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.424 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:89929 3.13 e125/106 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head v 125'167, mlcod=119'165)", 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.677012822000002, 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047142678, 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:38.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714501+0000", 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714502+0000", 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000001e-07 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728795+0000", 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014293337999999999 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728817+0000", 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1708e-05 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728831+0000", 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4239e-05 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728938+0000", 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107114 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.728964+0000", 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5513000000000001e-05 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761571+0000", 2024-04-24T00:36:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032606701000000002 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.761644+0000", 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3615999999999994e-05 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105170 3.16 e152/112 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head v 152'511, mlcod=150'509)", 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.702319158000002, 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022303394000000001, 2024-04-24T00:36:38.429 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689195+0000", 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689196+0000", 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7399999999999991e-07 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692342+0000", 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031461710000000001 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692378+0000", 2024-04-24T00:36:38.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5707999999999998e-05 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692396+0000", 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8110000000000001e-05 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692483+0000", 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6565999999999998e-05 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692513+0000", 2024-04-24T00:36:38.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0151999999999999e-05 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711444+0000", 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018930827000000001 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711499+0000", 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4886000000000003e-05 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105172 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [write 0~917504 [fadvise_sequential+fadvise_nocache],zero 917504~393216,write 1310720~131072 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.701679903999999, 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024973104999999999, 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105172 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689834+0000", 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.689837+0000", 2024-04-24T00:36:38.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.429e-06 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693186+0000", 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033495899999999999 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693214+0000", 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7616000000000001e-05 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693229+0000", 2024-04-24T00:36:38.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5424999999999999e-05 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693350+0000", 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120711 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693534+0000", 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018363200000000001 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:38.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.693878+0000", 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000343958 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.712994+0000", 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019116371 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714688+0000", 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016935539999999999 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714754+0000", 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6297000000000006e-05 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.714807+0000", 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3522e-05 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105169 3.1f e152/59 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head v 152'697, mlcod=152'693)", 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.699000473000002, 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041348306000000001, 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692514+0000", 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.692515+0000", 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699647+0000", 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071313169999999999 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699676+0000", 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9876999999999999e-05 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699693+0000", 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6973000000000002e-05 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699779+0000", 2024-04-24T00:36:38.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5169999999999999e-05 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.699805+0000", 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6120000000000001e-05 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733817+0000", 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034012112999999997 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733862+0000", 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5252e-05 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.150815968, 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:36:38.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:36:38.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:36:38.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.121048486999999, 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.453 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109747 3.1f 3:fa7fa4bf:::rbd_data.115edc839ccf.000000000000006c:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.618720059999999, 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016743020000000001, 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109747 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772794+0000", 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772795+0000", 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:36:38.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772799+0000", 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.391e-06 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772811+0000", 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1693e-05 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772821+0000", 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.004e-05 2024-04-24T00:36:38.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777269+0000", 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044474149999999997 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777464+0000", 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195353 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.789364+0000", 2024-04-24T00:36:38.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011900617 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.789537+0000", 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017274900000000001 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109751 3.1f 3:faff7380:::rbd_data.115edc839ccf.0000000000000070:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.618679315, 2024-04-24T00:36:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017461602, 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109751 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:38.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772835+0000", 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772836+0000", 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772839+0000", 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.817e-06 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772851+0000", 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1378e-05 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772860+0000", 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6750000000000008e-06 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777515+0000", 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046550200000000002 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.777744+0000", 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022927399999999999 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.790180+0000", 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012436107 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.790297+0000", 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011663 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109761 3.12 3:4d42c299:::rbd_data.115edc839ccf.000000000000007a:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.618642874000001, 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015851832999999999, 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109761 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772871+0000", 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772872+0000", 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7100000000000001e-07 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772876+0000", 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8139999999999998e-06 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772888+0000", 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1833e-05 2024-04-24T00:36:38.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772897+0000", 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4390000000000006e-06 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775136+0000", 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022391020000000002 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:38.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775322+0000", 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018560199999999999 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788599+0000", 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013277512999999999 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788723+0000", 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012385900000000001 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109770 3.18 3:1c37082e:::rbd_data.115edc839ccf.0000000000000083:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.618565461999999, 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020896860999999999, 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109770 2024-04-24T00:36:38.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772949+0000", 2024-04-24T00:36:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5899999999999996e-07 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772953+0000", 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9199999999999997e-06 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772964+0000", 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0964999999999999e-05 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772973+0000", 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9679999999999995e-06 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.781771+0000", 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0087973119999999998 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.781948+0000", 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000177398 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.793712+0000", 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011763852999999999 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.793846+0000", 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133786 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109775 3.18 3:1bef16aa:::rbd_data.115edc839ccf.0000000000000088:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:38.472 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.618528572000001, 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021323353999999999, 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109775 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772986+0000", 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6400000000000002e-07 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.772990+0000", 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.647e-06 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773001+0000", 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1399999999999999e-05 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773011+0000", 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2699999999999993e-06 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.782127+0000", 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091162650000000001 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.782286+0000", 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015908 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.794246+0000", 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011959722000000001 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.794309+0000", 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3305999999999993e-05 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:109779 3.12 3:48246b4e:::rbd_data.115edc839ccf.000000000000008c:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.618492986, 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015522284000000001, 2024-04-24T00:36:38.477 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109779 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773021+0000", 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773022+0000", 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.37e-07 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773025+0000", 2024-04-24T00:36:38.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773036+0000", 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0635000000000001e-05 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.773045+0000", 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6819999999999992e-06 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775372+0000", 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023270669999999999 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.775551+0000", 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017971 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788416+0000", 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012864123999999999 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:24.788544+0000", 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012804 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:122724 3.0 3:06b99ede:::rbd_data.115e1702e7a8.000000000000006a:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e168)", 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.173217+0000", 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2182973959999996, 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015946536000000001, 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 122724 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.173217+0000", 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.173217+0000", 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.173217+0000", 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2899999999999999e-07 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.173220+0000", 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.187e-06 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.173226+0000", 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4250000000000003e-06 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.173231+0000", 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.096e-06 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.188138+0000", 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014906772 2024-04-24T00:36:38.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.188221+0000", 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3479000000000005e-05 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.189163+0000", 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094214799999999997 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:122812 3.0 3:0497a7c0:::rbd_data.115e1702e7a8.00000000000000c2:head [sparse-read 0~163840] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e168)", 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.175359+0000", 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2161557759999999, 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016488342, 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 122812 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.175359+0000", 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.175359+0000", 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.175359+0000", 2024-04-24T00:36:38.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6100000000000001e-07 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.175362+0000", 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2710000000000001e-06 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.175371+0000", 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4810000000000006e-06 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.175377+0000", 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3589999999999996e-06 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.189371+0000", 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013993830000000001 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.189458+0000", 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7150000000000004e-05 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.191847+0000", 2024-04-24T00:36:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023886900000000002 2024-04-24T00:36:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.491 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:38.492 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:38.492 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:38.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:38.494 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:38.494 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:38.494 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:38.494 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:38.495 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:38.495 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:38.496 INFO:teuthology.orchestra.run.smithi064.stdout:4602 read 0xff566e thru 0xffc7d8 (0x716b bytes) 2024-04-24T00:36:38.496 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:38.487+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:38.525 INFO:teuthology.orchestra.run.smithi064.stdout:4603 write 0xed20971 thru 0xed21ad7 (0x1167 bytes) 2024-04-24T00:36:38.544 INFO:teuthology.orchestra.run.smithi064.stdout:4604 trunc from 0xed21ad8 to 0x5be71b1 2024-04-24T00:36:38.565 INFO:teuthology.orchestra.run.smithi064.stdout:4605 trunc from 0x5be71b1 to 0x78a7403 2024-04-24T00:36:38.580 INFO:teuthology.orchestra.run.smithi064.stdout:4606 trunc from 0x78a7403 to 0x86541d1 2024-04-24T00:36:38.593 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:38.591+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:38.594 INFO:teuthology.orchestra.run.smithi064.stdout:4607 write 0x45151b8 thru 0x4518c32 (0x3a7b bytes) 2024-04-24T00:36:38.600 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:38.619 INFO:teuthology.orchestra.run.smithi064.stdout:4610 writesame 0xb7da6a4 thru 0xb7e7420 data_size 0x69(0xcd7d bytes) 2024-04-24T00:36:38.619 INFO:teuthology.orchestra.run.smithi064.stdout:4611 read 0x10bad70 thru 0x10ca466 (0xf6f7 bytes) 2024-04-24T00:36:38.619 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:38.637 INFO:teuthology.orchestra.run.smithi064.stdout:4614 trunc from 0xb7e7421 to 0x751514 2024-04-24T00:36:38.660 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:38.660 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:38.660 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:36:38.660 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:38.668 INFO:teuthology.orchestra.run.smithi064.stdout:4615 punch from 0xc609d to 0xcaf32, (0x4e95 bytes) 2024-04-24T00:36:38.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:38.698 INFO:teuthology.orchestra.run.smithi064.stdout:4616 write 0x9b8fb9a thru 0x9b94e7f (0x52e6 bytes) 2024-04-24T00:36:38.699 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:38.695+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:38.704 INFO:teuthology.orchestra.run.smithi064.stdout:4618 write 0x1fa0c4a thru 0x1fa2126 (0x14dd bytes) 2024-04-24T00:36:38.728 INFO:teuthology.orchestra.run.smithi064.stdout:4619 writesame 0xb52a239 thru 0xb52ee77 data_size 0x1(0x4c3f bytes) 2024-04-24T00:36:38.740 INFO:teuthology.orchestra.run.smithi064.stdout:4620 trunc from 0xb52ee78 to 0x35393ba 2024-04-24T00:36:38.795 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:38.791+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:38.800 INFO:teuthology.orchestra.run.smithi064.stdout:4621 write 0xea01b18 thru 0xea0e4e4 (0xc9cd bytes) 2024-04-24T00:36:38.808 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:38.809 INFO:teuthology.orchestra.run.smithi064.stdout:4623 writesame 0x3eb3c35 thru 0x3ebc973 data_size 0x11(0x8d3f bytes) 2024-04-24T00:36:38.818 INFO:teuthology.orchestra.run.smithi064.stdout:4625 writesame 0x4fa6428 thru 0x4fb28f5 data_size 0x36(0xc4ce bytes) 2024-04-24T00:36:38.830 INFO:teuthology.orchestra.run.smithi064.stdout:4626 write 0x1425dbd thru 0x1435936 (0xfb7a bytes) 2024-04-24T00:36:38.833 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:38.833 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:36:38.834 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:36:38.834 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:36:38.834 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:38.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:38.852 INFO:teuthology.orchestra.run.smithi064.stdout:4628 trunc from 0xea0e4e5 to 0xc41d510 2024-04-24T00:36:38.879 INFO:teuthology.orchestra.run.smithi064.stdout:4629 read 0xba952ff thru 0xba9d344 (0x8046 bytes) 2024-04-24T00:36:38.879 INFO:teuthology.orchestra.run.smithi064.stdout:4630 write 0x2501ef6 thru 0x25093af (0x74ba bytes) 2024-04-24T00:36:38.888 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:38.889 INFO:teuthology.orchestra.run.smithi064.stdout:4632 read 0xef0e81 thru 0xef35d2 (0x2752 bytes) 2024-04-24T00:36:38.889 INFO:teuthology.orchestra.run.smithi064.stdout:4635 writesame 0xba4f0a5 thru 0xba5837f data_size 0x49(0x92db bytes) 2024-04-24T00:36:38.895 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:38.891+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:38.897 INFO:teuthology.orchestra.run.smithi064.stdout:4636 read 0xeca185 thru 0xed8be8 (0xea64 bytes) 2024-04-24T00:36:38.899 INFO:teuthology.orchestra.run.smithi064.stdout:4638 trunc from 0xc41d510 to 0xcc0b96d 2024-04-24T00:36:38.911 INFO:teuthology.orchestra.run.smithi064.stdout:4640 clone 33 order 22 su 131072 sc 9 2024-04-24T00:36:38.986 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.770811159, 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.992 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.77046024399999, 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:38.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:38.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:38.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:38.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:38.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:38.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.76357192399999, 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:38.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:39.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.505290298000006, 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:39.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:39.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.495013881999995, 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:39.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:39.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.033063079000001, 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:39.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.018 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.020654812999993, 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:39.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.020623059000002, 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:39.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.017090562999996, 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:39.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:39.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.007029058000001, 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.793086150999997, 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:39.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:39.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:39.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.783074206999999, 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:39.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:39.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:39.048 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.509695344999997, 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.493026896000003, 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:39.054 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.307129113000002, 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:39.058 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:39.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:39.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.623370756, 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.601153758000002, 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:39.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.439932478000003, 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:39.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:39.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.29658392, 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:39.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:39.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:36:39.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.28395896, 2024-04-24T00:36:39.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:39.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:39.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:39.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:39.090 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:39.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:39.092 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:39.087+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:39.192 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:39.187+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:39.222 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:39.222 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:39.222 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:39.222 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:39.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:39.293 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:39.287+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:39.358 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:39.358 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:39.358 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:39.358 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:39.358 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:39.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:39.394 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:39.391+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:39.495 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:39.491+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:39.498 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:39.498 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:39.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:39.498 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:39.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.279919398, 2024-04-24T00:36:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:39.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:39.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:39.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 131.85813370400001, 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:39.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:39.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.716006137999997, 2024-04-24T00:36:39.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:39.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.578234355999996, 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.517 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.540752037999994, 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.523 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:39.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:39.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:39.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.511423076, 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:36:39.529 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.530 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:36:39.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:36:39.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.314161173000002, 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.536 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:39.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:39.541 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.878406173999998, 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.542 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:39.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:39.547 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.868548138000001, 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.548 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:39.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:39.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.848198529999998, 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.554 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:39.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:39.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.814153195000003, 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:39.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:39.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.811843359000001, 2024-04-24T00:36:39.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:39.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:39.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.803274559999998, 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.570 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:39.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.864125467000001, 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.576 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:39.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.974735471000002, 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:39.582 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:39.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:39.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:39.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.931259674000003, 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:39.593 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:39.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.957286371999999, 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:39.597 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:39.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:39.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:39.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.938144469000001, 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.606 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.978856916000002, 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:39.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:39.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:39.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:39.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:39.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.944854494000001, 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:39.612 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:39.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:39.616 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:39.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:39.618 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:39.615+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:39.719 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:39.715+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:39.796 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:39.797 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:39.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:39.857 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:39.855+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:39.920 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:39.915+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:39.930 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:39.930 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:39.930 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:39.930 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:39.930 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:39.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:39.964 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5, 0] out_osds: [7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:39.964 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:39.964 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-24T00:36:39.964 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:36:39.964 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:36:39.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-24T00:36:39.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:36:40.033 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:40.031+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:40.108 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:40.108 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "" 2024-04-24T00:36:40.108 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:40.112 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.40036217900001, 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:36:40.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:40.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:40.118 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.40026425400001, 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:40.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:36:40.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:36:40.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:36:40.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 119.040884722, 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:40.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.130 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 103.217900735, 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.131 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:36:40.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.630140596000004, 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:36:40.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:36:40.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:40.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:36:40.142 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.604871990000007, 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:40.146 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.751668269999996, 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:36:40.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:40.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.152 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.719063156000004, 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.314488462, 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:36:40.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:36:40.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:36:40.162 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.256648232000003, 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89931 3.1f 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head [write 0~4577672 [fadvise_sequential+fadvise_nocache],zero 4718592~262144,zero 6291456~179747,write 6471203~13759 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.417115629000001, 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068626284999999995, 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.166 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89931 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691618+0000", 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2949999999999998e-06 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704054+0000", 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012436665 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704090+0000", 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5778999999999997e-05 2024-04-24T00:36:40.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704107+0000", 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7122999999999999e-05 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704255+0000", 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014735099999999999 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704588+0000", 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033309499999999999 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.705598+0000", 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001010597 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.758968+0000", 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053369708000000002 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760090+0000", 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011214840000000001 2024-04-24T00:36:40.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760173+0000", 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3147999999999997e-05 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760241+0000", 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8040000000000006e-05 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:36:40.171 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.415085883000003, 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:36:40.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:36:40.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90259 3.2 3:4377c456:::rbd_data.115e56aa02fc.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.578602717999999, 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064637350999999996, 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.176 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90259 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530129+0000", 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530137+0000", 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530152+0000", 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4612e-05 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530160+0000", 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-06 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530341+0000", 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018113600000000001 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530535+0000", 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193606 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.534329+0000", 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037947839999999998 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594447+0000", 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060117469999999999 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594580+0000", 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013329699999999999 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594679+0000", 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9085000000000004e-05 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594765+0000", 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5533999999999998e-05 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.436338331999998, 2024-04-24T00:36:40.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:36:40.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.406696279000002, 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:36:40.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.190 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.260180867999999, 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.191 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:36:40.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:36:40.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.471408310000001, 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:40.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.448264897000001, 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:40.207 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.574956148999998, 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:40.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:36:40.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.522273872, 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.213 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:36:40.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:36:40.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:36:40.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.217 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.217 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.217 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:40.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:40.219 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:40.215+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:40.319 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:40.315+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:40.390 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:40.390 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:40.390 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:40.390 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:40.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:40.449 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:40.447+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:40.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:40.521 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:40.515+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:40.574 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:36:40.575 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:40.575 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:40.594 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:40.622 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:40.619+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:40.662 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:40.663 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:40.680 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:40.680 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:40.697 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:40.697 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:40.714 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:40.715 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:40.723 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:40.719+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:40.734 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:40.735 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:40.752 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-04-24T00:36:40.755 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:40.755 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:40.774 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:40.774 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:40.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:40.774 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 142.23506567800001, 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:36:40.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.780 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 142.20102862499999, 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.781 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30152 3.a 3:54c8e8b2:::rbd_data.115e8210e51a.0000000000000014:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:40.785 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 119.61030266100001, 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048903929999999998, 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30152 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157017+0000", 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157020+0000", 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0869999999999998e-06 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157033+0000", 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2906e-05 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157070+0000", 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.693e-05 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157091+0000", 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0440000000000001e-05 2024-04-24T00:36:40.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157178+0000", 2024-04-24T00:36:40.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7313999999999998e-05 2024-04-24T00:36:40.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.157284+0000", 2024-04-24T00:36:40.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106149 2024-04-24T00:36:40.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:40.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.159470+0000", 2024-04-24T00:36:40.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021862520000000001 2024-04-24T00:36:40.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205620+0000", 2024-04-24T00:36:40.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046149297999999998 2024-04-24T00:36:40.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205723+0000", 2024-04-24T00:36:40.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103189 2024-04-24T00:36:40.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205832+0000", 2024-04-24T00:36:40.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108739 2024-04-24T00:36:40.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205921+0000", 2024-04-24T00:36:40.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9625999999999999e-05 2024-04-24T00:36:40.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.794 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.794 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30152 3.a e66/21 ondisk, result = 0)", 2024-04-24T00:36:40.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 119.601046536, 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039662068000000002, 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:36:40.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166273+0000", 2024-04-24T00:36:40.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166275+0000", 2024-04-24T00:36:40.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4839999999999999e-06 2024-04-24T00:36:40.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166280+0000", 2024-04-24T00:36:40.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6810000000000003e-06 2024-04-24T00:36:40.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166297+0000", 2024-04-24T00:36:40.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6762e-05 2024-04-24T00:36:40.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.166307+0000", 2024-04-24T00:36:40.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011e-05 2024-04-24T00:36:40.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205693+0000", 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039385186000000003 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205711+0000", 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8046999999999998e-05 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.205935+0000", 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022479800000000001 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.085914629, 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:36:40.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:36:40.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.058464668, 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:36:40.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:36:40.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:81511 3.d e111/106 3:b780edbf:::rbd_data.115e6f07caf6.0000000000000006:head v 111'118, mlcod=0'0)", 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.803468350000003, 2024-04-24T00:36:40.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029826869999999998, 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.963852+0000", 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.65e-07 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966954+0000", 2024-04-24T00:36:40.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031022379999999998 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966976+0000", 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1565000000000001e-05 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.966988+0000", 2024-04-24T00:36:40.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2428e-05 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967105+0000", 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011725799999999999 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.967132+0000", 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6863999999999999e-05 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993615+0000", 2024-04-24T00:36:40.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026482946 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.993678+0000", 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3206000000000004e-05 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84130 3.11 e111/106 3:8ac0652c:::rbd_data.115e6f07caf6.000000000000000b:head v 111'344, mlcod=108'342)", 2024-04-24T00:36:40.816 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.596925322999994, 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037411427999999997, 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.170395+0000", 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0699999999999999e-07 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171887+0000", 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00149255 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171909+0000", 2024-04-24T00:36:40.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1270000000000001e-05 2024-04-24T00:36:40.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.171918+0000", 2024-04-24T00:36:40.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6139999999999998e-06 2024-04-24T00:36:40.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172147+0000", 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022887100000000001 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.172341+0000", 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019398700000000001 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207639+0000", 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035298445999999997 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.207806+0000", 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016648300000000001 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:84132 3.14 e111/106 3:2c6d4ac0:::rbd_data.115e6f07caf6.000000000000000d:head v 111'167, mlcod=0'0)", 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.587091462999993, 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029744291999999999, 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180228+0000", 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180230+0000", 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8029999999999999e-06 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.184974+0000", 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047440540000000002 2024-04-24T00:36:40.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185003+0000", 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8860000000000002e-05 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185021+0000", 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7473999999999999e-05 2024-04-24T00:36:40.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185169+0000", 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148305 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.185196+0000", 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7444e-05 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209925+0000", 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024728208000000002 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.209973+0000", 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8143999999999998e-05 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:36:40.826 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 63.138392836000001, 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:40.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:36:40.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 63.132121578000003, 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.831 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.837 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 63.120666991999997, 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:40.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:40.842 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 63.066214447999997, 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:36:40.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:36:40.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105171 3.1e 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head [zero 0~262144,write 262144~131072 [fadvise_sequential+fadvise_nocache],zero 393216~131072,write 524288~917504 [fadvise_sequential+fadvise_nocache],zero 1441792~393216,write 1835008~131072 [fadvise_sequential+fadvise_nocache],write 2097152~917504 [fadvise_sequential+fadvise_nocache],zero 3014656~393216,write 3407872~131072 [fadvise_sequential+fadvise_nocache],zero 3538944~131072,write 3670016~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:40.847 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.076463956000001, 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061400402, 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105171 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690858+0000", 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5059999999999998e-06 2024-04-24T00:36:40.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697752+0000", 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006893766 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697795+0000", 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2589000000000002e-05 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697810+0000", 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5620999999999998e-05 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697936+0000", 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012529199999999999 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698150+0000", 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021430200000000001 2024-04-24T00:36:40.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698445+0000", 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029468099999999998 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736712+0000", 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038267781000000001 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752153+0000", 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015441043999999999 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752204+0000", 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752256+0000", 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2286e-05 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105174 3.1e 3:7c3077cb:::rbd_data.115ec2448e7a.0000000000000005:head [write 0~393216 [fadvise_sequential+fadvise_nocache],zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache],write 1048576~917504 [fadvise_sequential+fadvise_nocache],zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache],zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache],zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.705256+0000", 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.062064341999999, 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047556550000000003, 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105174 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.705256+0000", 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.705256+0000", 2024-04-24T00:36:40.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.705257+0000", 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9369999999999998e-06 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.707892+0000", 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002634548 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.707909+0000", 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7357999999999999e-05 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.707918+0000", 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7350000000000002e-06 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.718620+0000", 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010702006999999999 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.718817+0000", 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019676399999999999 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.719182+0000", 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036548899999999998 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736720+0000", 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017538061000000001 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752746+0000", 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016025463 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752786+0000", 2024-04-24T00:36:40.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.053e-05 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752812+0000", 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5658000000000001e-05 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:109782 3.10 3:09d76ef5:::rbd_data.115edc839ccf.000000000000008f:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:40.860 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:24.773981+0000", 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.993339291, 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032176689000000001, 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 109782 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.773981+0000", 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.773981+0000", 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.773982+0000", 2024-04-24T00:36:40.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0330000000000001e-06 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.773987+0000", 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5099999999999998e-06 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.774004+0000", 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.668e-05 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.774017+0000", 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2977e-05 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.797411+0000", 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023394074000000001 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:40.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.797770+0000", 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035946399999999999 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.805759+0000", 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079887940000000005 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.806157+0000", 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000398157 2024-04-24T00:36:40.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:109847 3.10 3:0d356d52:::rbd_data.115edc839ccf.00000000000000d0:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:24.774401+0000", 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.992919118, 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024391771, 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:40.866 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 109847 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.774401+0000", 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.774401+0000", 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.774402+0000", 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.124e-06 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.774437+0000", 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5036000000000001e-05 2024-04-24T00:36:40.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.774454+0000", 2024-04-24T00:36:40.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7350000000000002e-05 2024-04-24T00:36:40.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.774469+0000", 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4643e-05 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.798556+0000", 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024087233 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.798793+0000", 2024-04-24T00:36:40.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023638500000000001 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:109925 3.10 3:098221ce:::rbd_data.115edc839ccf.000000000000011e:head [sparse-read 0~262144] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:24.775598+0000", 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 15.991722013, 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024539465999999999, 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.871 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:40.872 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.873 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.873 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.873 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 109925 2024-04-24T00:36:40.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.873 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.775598+0000", 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.775598+0000", 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.775599+0000", 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2529999999999999e-06 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.775605+0000", 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3269999999999997e-06 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.776459+0000", 2024-04-24T00:36:40.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00085360899999999997 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.776525+0000", 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5944000000000004e-05 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.799882+0000", 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023356597999999999 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:24.800137+0000", 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025573499999999999 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:124553 3.11 e174/106 3:8a910f44:::rbd_data.115ee44a9909.0000000000000001:head v 174'428, mlcod=174'425)", 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:39.649160+0000", 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.118159831, 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030044919999999999, 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.877 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.649160+0000", 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.649160+0000", 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.649162+0000", 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.629e-06 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.650468+0000", 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001305896 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.650502+0000", 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4365e-05 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.650522+0000", 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9656999999999998e-05 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.650638+0000", 2024-04-24T00:36:40.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011644799999999999 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.650684+0000", 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5692000000000002e-05 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.679140+0000", 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028456162 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.679205+0000", 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5071000000000001e-05 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:124575 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:head [write 0~786432 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:39.649890+0000", 2024-04-24T00:36:40.882 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.117430148, 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025648699, 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 124575 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.649890+0000", 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.649890+0000", 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.649894+0000", 2024-04-24T00:36:40.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1919999999999998e-06 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.651332+0000", 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014377070000000001 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.651360+0000", 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8702000000000001e-05 2024-04-24T00:36:40.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.651377+0000", 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6844999999999999e-05 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.653348+0000", 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00197058 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.653624+0000", 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027673299999999998 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.653927+0000", 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000302288 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:40.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.668424+0000", 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014497197 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.675320+0000", 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068965279999999999 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.675441+0000", 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120581 2024-04-24T00:36:40.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:39.675538+0000", 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7345999999999999e-05 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:40.889 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:40.890 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:40.891 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:40.892 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:40.892 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:40.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:40.893 INFO:teuthology.orchestra.run.smithi064.stdout:4641 read 0xa51cf71 thru 0xa51f4d7 (0x2567 bytes) 2024-04-24T00:36:40.893 INFO:teuthology.orchestra.run.smithi064.stdout:4642 punch from 0xa8f77e0 to 0xa903d98, (0xc5b8 bytes) 2024-04-24T00:36:40.895 INFO:teuthology.orchestra.run.smithi064.stdout:4644 trunc from 0xcc0b96d to 0xe598295 2024-04-24T00:36:40.895 INFO:teuthology.orchestra.run.smithi064.stdout:4645 read 0xb69ebc0 thru 0xb6abbdf (0xd020 bytes) 2024-04-24T00:36:40.895 INFO:teuthology.orchestra.run.smithi064.stdout:4648 write 0x9b4b81c thru 0x9b4c4c4 (0xca9 bytes) 2024-04-24T00:36:40.895 INFO:teuthology.orchestra.run.smithi064.stdout:4649 write 0xee09b75 thru 0xee0f4b7 (0x5943 bytes) 2024-04-24T00:36:40.895 INFO:teuthology.orchestra.run.smithi064.stdout:4650 write 0xc136797 thru 0xc143e2c (0xd696 bytes) 2024-04-24T00:36:40.895 INFO:teuthology.orchestra.run.smithi064.stdout:4651 read 0xe108482 thru 0xe114177 (0xbcf6 bytes) 2024-04-24T00:36:40.896 INFO:teuthology.orchestra.run.smithi064.stdout:4652 writesame 0xc4ed580 thru 0xc4f1e69 data_size 0x66(0x48ea bytes) 2024-04-24T00:36:40.896 INFO:teuthology.orchestra.run.smithi064.stdout:4653 writesame 0x5e95a9e thru 0x5e992fd data_size 0x2c(0x3860 bytes) 2024-04-24T00:36:40.896 INFO:teuthology.orchestra.run.smithi064.stdout:4654 writesame 0xcfbd1d9 thru 0xcfc1930 data_size 0x1(0x4758 bytes) 2024-04-24T00:36:40.896 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:40.887+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:40.905 INFO:teuthology.orchestra.run.smithi064.stdout:4655 read 0x2554d64 thru 0x2558bd6 (0x3e73 bytes) 2024-04-24T00:36:40.905 INFO:teuthology.orchestra.run.smithi064.stdout:4657 punch from 0x9ae9894 to 0x9af42a3, (0xaa0f bytes) 2024-04-24T00:36:40.924 INFO:teuthology.orchestra.run.smithi064.stdout:4658 trunc from 0xee0f4b8 to 0x30a146f 2024-04-24T00:36:40.988 INFO:teuthology.orchestra.run.smithi064.stdout:4660 write 0xa5c74a6 thru 0xa5c81b0 (0xd0b bytes) 2024-04-24T00:36:40.991 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:40.987+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:40.996 INFO:teuthology.orchestra.run.smithi064.stdout:4661 punch from 0x4b9a336 to 0x4b9f776, (0x5440 bytes) 2024-04-24T00:36:41.018 INFO:teuthology.orchestra.run.smithi064.stdout:4662 trunc from 0xa5c81b1 to 0x20d2b4b 2024-04-24T00:36:41.072 INFO:teuthology.orchestra.run.smithi064.stdout:4663 write 0x44dae7a thru 0x44e2df9 (0x7f80 bytes) 2024-04-24T00:36:41.073 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:41.073 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:41.082 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.083 INFO:teuthology.orchestra.run.smithi064.stdout:4667 write 0x2c55550 thru 0x2c62b0e (0xd5bf bytes) 2024-04-24T00:36:41.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:41.092 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:41.087+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:41.122 INFO:teuthology.orchestra.run.smithi064.stdout:4668 write 0x91ede20 thru 0x91f4f75 (0x7156 bytes) 2024-04-24T00:36:41.158 INFO:teuthology.orchestra.run.smithi064.stdout:4672 writesame 0xe9e5fff thru 0xe9f1624 data_size 0x5(0xb626 bytes) 2024-04-24T00:36:41.158 INFO:teuthology.orchestra.run.smithi064.stdout:4673 write 0x118641a thru 0x118ab21 (0x4708 bytes) 2024-04-24T00:36:41.162 INFO:teuthology.orchestra.run.smithi064.stdout:4674 read 0xe5ac101 thru 0xe5b2c76 (0x6b76 bytes) 2024-04-24T00:36:41.162 INFO:teuthology.orchestra.run.smithi064.stdout:4675 punch from 0x94422fd to 0x9445456, (0x3159 bytes) 2024-04-24T00:36:41.162 INFO:teuthology.orchestra.run.smithi064.stdout:4676 read 0x36c5a9f thru 0x36cf461 (0x99c3 bytes) 2024-04-24T00:36:41.163 INFO:teuthology.orchestra.run.smithi064.stdout:4678 punch from 0x6cf4e97 to 0x6cf8529, (0x3692 bytes) 2024-04-24T00:36:41.163 INFO:teuthology.orchestra.run.smithi064.stdout:4681 write 0x8d08801 thru 0x8d0a5ea (0x1dea bytes) 2024-04-24T00:36:41.169 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.169 INFO:teuthology.orchestra.run.smithi064.stdout:4683 read 0x8371aa8 thru 0x837ea3e (0xcf97 bytes) 2024-04-24T00:36:41.169 INFO:teuthology.orchestra.run.smithi064.stdout:4684 write 0x477a840 thru 0x478026f (0x5a30 bytes) 2024-04-24T00:36:41.175 INFO:teuthology.orchestra.run.smithi064.stdout:4685 write 0x8a710d2 thru 0x8a80549 (0xf478 bytes) 2024-04-24T00:36:41.190 INFO:teuthology.orchestra.run.smithi064.stdout:4686 write 0x3f9a1cd thru 0x3fa303b (0x8e6f bytes) 2024-04-24T00:36:41.192 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:41.187+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:41.196 INFO:teuthology.orchestra.run.smithi064.stdout:4687 writesame 0x1335cba thru 0x1335e63 data_size 0x3(0x1aa bytes) 2024-04-24T00:36:41.237 INFO:teuthology.orchestra.run.smithi064.stdout:4689 trunc from 0xe9f1625 to 0x7f54c6 2024-04-24T00:36:41.261 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:41.262 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:41.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:41.283 INFO:teuthology.orchestra.run.smithi064.stdout:4690 write 0x2efa2a5 thru 0x2f08b07 (0xe863 bytes) 2024-04-24T00:36:41.293 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:41.287+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:41.311 INFO:teuthology.orchestra.run.smithi064.stdout:4691 trunc from 0x2f08b08 to 0xb4e1d2b 2024-04-24T00:36:41.322 INFO:teuthology.orchestra.run.smithi064.stdout:4692 read 0x1a13f2 thru 0x1ad1c0 (0xbdcf bytes) 2024-04-24T00:36:41.324 INFO:teuthology.orchestra.run.smithi064.stdout:4694 read 0x60bf538 thru 0x60c9760 (0xa229 bytes) 2024-04-24T00:36:41.331 INFO:teuthology.orchestra.run.smithi064.stdout:4696 trunc from 0xb4e1d2b to 0xe5df12b 2024-04-24T00:36:41.343 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.343 INFO:teuthology.orchestra.run.smithi064.stdout:4700 write 0x1611411 thru 0x161bc6e (0xa85e bytes) 2024-04-24T00:36:41.349 INFO:teuthology.orchestra.run.smithi064.stdout:4701 write 0x82607e2 thru 0x826c9eb (0xc20a bytes) 2024-04-24T00:36:41.361 INFO:teuthology.orchestra.run.smithi064.stdout:4702 write 0xbf73845 thru 0xbf798ef (0x60ab bytes) 2024-04-24T00:36:41.371 INFO:teuthology.orchestra.run.smithi064.stdout:4703 punch from 0x1bc32ed to 0x1bc87dd, (0x54f0 bytes) 2024-04-24T00:36:41.371 INFO:teuthology.orchestra.run.smithi064.stdout:4704 writesame 0x8e4b11e thru 0x8e4ddd9 data_size 0xe(0x2cbc bytes) 2024-04-24T00:36:41.378 INFO:teuthology.orchestra.run.smithi064.stdout:4705 read 0x1f6a523 thru 0x1f7151a (0x6ff8 bytes) 2024-04-24T00:36:41.380 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.380 INFO:teuthology.orchestra.run.smithi064.stdout:4707 read 0x2b38449 thru 0x2b3aeb3 (0x2a6b bytes) 2024-04-24T00:36:41.380 INFO:teuthology.orchestra.run.smithi064.stdout:4708 read 0x2cfa3e9 thru 0x2cfdcf6 (0x390e bytes) 2024-04-24T00:36:41.380 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.380 INFO:teuthology.orchestra.run.smithi064.stdout:4710 read 0x79b18ba thru 0x79be4d6 (0xcc1d bytes) 2024-04-24T00:36:41.380 INFO:teuthology.orchestra.run.smithi064.stdout:4711 read 0x257cc3e thru 0x257ea60 (0x1e23 bytes) 2024-04-24T00:36:41.380 INFO:teuthology.orchestra.run.smithi064.stdout:4713 punch from 0x194ea9c to 0x194f134, (0x698 bytes) 2024-04-24T00:36:41.380 INFO:teuthology.orchestra.run.smithi064.stdout:4715 write 0xad98780 thru 0xad9bf57 (0x37d8 bytes) 2024-04-24T00:36:41.387 INFO:teuthology.orchestra.run.smithi064.stdout:4716 write 0x816544d thru 0x81695e1 (0x4195 bytes) 2024-04-24T00:36:41.394 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:41.391+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:41.396 INFO:teuthology.orchestra.run.smithi064.stdout:4717 read 0xdc8cba6 thru 0xdc92017 (0x5472 bytes) 2024-04-24T00:36:41.396 INFO:teuthology.orchestra.run.smithi064.stdout:4718 punch from 0x8bf8484 to 0x8c044a7, (0xc023 bytes) 2024-04-24T00:36:41.397 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.397 INFO:teuthology.orchestra.run.smithi064.stdout:4720 punch from 0x9cacb37 to 0x9cad725, (0xbee bytes) 2024-04-24T00:36:41.397 INFO:teuthology.orchestra.run.smithi064.stdout:4722 write 0xbb4e823 thru 0xbb5c296 (0xda74 bytes) 2024-04-24T00:36:41.405 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:41.406 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:41.411 INFO:teuthology.orchestra.run.smithi064.stdout:4723 write 0x185ad80 thru 0x18605b8 (0x5839 bytes) 2024-04-24T00:36:41.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:41.423 INFO:teuthology.orchestra.run.smithi064.stdout:4726 trunc from 0xe5df12b to 0xc54f050 2024-04-24T00:36:41.437 INFO:teuthology.orchestra.run.smithi064.stdout:4728 read 0xaa07baf thru 0xaa1290a (0xad5c bytes) 2024-04-24T00:36:41.437 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.437 INFO:teuthology.orchestra.run.smithi064.stdout:4730 write 0xa33c785 thru 0xa33f48c (0x2d08 bytes) 2024-04-24T00:36:41.454 INFO:teuthology.orchestra.run.smithi064.stdout:4732 write 0xca789fa thru 0xca79ab3 (0x10ba bytes) 2024-04-24T00:36:41.458 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.459 INFO:teuthology.orchestra.run.smithi064.stdout:4735 trunc from 0xca79ab4 to 0xc89f899 2024-04-24T00:36:41.475 INFO:teuthology.orchestra.run.smithi064.stdout:4736 punch from 0x741641 to 0x747d19, (0x66d8 bytes) 2024-04-24T00:36:41.475 INFO:teuthology.orchestra.run.smithi064.stdout:4737 punch from 0xa2f423b to 0xa3030d5, (0xee9a bytes) 2024-04-24T00:36:41.491 INFO:teuthology.orchestra.run.smithi064.stdout:4739 trunc from 0xc89f899 to 0x3b95bd8 2024-04-24T00:36:41.494 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:41.491+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:41.537 INFO:teuthology.orchestra.run.smithi064.stdout:4741 writesame 0x4adeec8 thru 0x4ae6100 data_size 0x51(0x7239 bytes) 2024-04-24T00:36:41.537 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.538 INFO:teuthology.orchestra.run.smithi064.stdout:4745 punch from 0x307b46e to 0x307e8eb, (0x347d bytes) 2024-04-24T00:36:41.538 INFO:teuthology.orchestra.run.smithi064.stdout:4746 read 0x449524c thru 0x449e3b1 (0x9166 bytes) 2024-04-24T00:36:41.538 INFO:teuthology.orchestra.run.smithi064.stdout:4750 punch from 0x17b4b0f to 0x17c0bbf, (0xc0b0 bytes) 2024-04-24T00:36:41.538 INFO:teuthology.orchestra.run.smithi064.stdout:4752 writesame 0x15503b6 thru 0x155c7d4 data_size 0x1(0xc41f bytes) 2024-04-24T00:36:41.560 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:41.561 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:41.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:41.595 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:41.591+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:41.696 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:41.691+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:41.716 INFO:teuthology.orchestra.run.smithi064.stdout:4753 writesame 0x980dfa2 thru 0x9815480 data_size 0x1(0x74df bytes) 2024-04-24T00:36:41.717 INFO:teuthology.orchestra.run.smithi064.stdout:4755 read 0x668037f thru 0x668c4c5 (0xc147 bytes) 2024-04-24T00:36:41.717 INFO:teuthology.orchestra.run.smithi064.stdout:4756 punch from 0xcccf26 to 0xcdb7a9, (0xe883 bytes) 2024-04-24T00:36:41.717 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.730 INFO:teuthology.orchestra.run.smithi064.stdout:4761 trunc from 0x9815481 to 0x39c335b 2024-04-24T00:36:41.750 INFO:teuthology.orchestra.run.smithi064.stdout:4762 punch from 0x3410dd3 to 0x34126e0, (0x190d bytes) 2024-04-24T00:36:41.750 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:41.750 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:41.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:41.775 INFO:teuthology.orchestra.run.smithi064.stdout:4763 write 0x720e85e thru 0x7219ac2 (0xb265 bytes) 2024-04-24T00:36:41.782 INFO:teuthology.orchestra.run.smithi064.stdout:4764 write 0x6cdae9 thru 0x6d73fb (0x9913 bytes) 2024-04-24T00:36:41.789 INFO:teuthology.orchestra.run.smithi064.stdout:4765 read 0x263aa4a thru 0x2648b85 (0xe13c bytes) 2024-04-24T00:36:41.789 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.790 INFO:teuthology.orchestra.run.smithi064.stdout:4768 read 0x5b7cd09 thru 0x5b8a7f8 (0xdaf0 bytes) 2024-04-24T00:36:41.797 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:41.791+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:4769 trunc from 0x7219ac3 to 0xc918f53 2024-04-24T00:36:41.813 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.813 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:41.813 INFO:teuthology.orchestra.run.smithi064.stdout:4775 trunc from 0xc918f53 to 0xc972e3e 2024-04-24T00:36:41.837 INFO:teuthology.orchestra.run.smithi064.stdout:4776 trunc from 0xc972e3e to 0x6cc2ec4 2024-04-24T00:36:41.861 INFO:teuthology.orchestra.run.smithi064.stdout:4777 trunc from 0x6cc2ec4 to 0x39eeb15 2024-04-24T00:36:41.886 INFO:teuthology.orchestra.run.smithi064.stdout:4778 read 0x27d2c6b thru 0x27d40e2 (0x1478 bytes) 2024-04-24T00:36:41.897 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:41.895+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:41.902 INFO:teuthology.orchestra.run.smithi064.stdout:4779 write 0x517c1c4 thru 0x518b77a (0xf5b7 bytes) 2024-04-24T00:36:41.911 INFO:teuthology.orchestra.run.smithi064.stdout:4780 read 0x4301d53 thru 0x430db4c (0xbdfa bytes) 2024-04-24T00:36:41.911 INFO:teuthology.orchestra.run.smithi064.stdout:4781 write 0x4894818 thru 0x48a008e (0xb877 bytes) 2024-04-24T00:36:41.912 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:41.913 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:41.924 INFO:teuthology.orchestra.run.smithi064.stdout:4782 trunc from 0x518b77b to 0x71f6927 2024-04-24T00:36:41.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:41.944 INFO:teuthology.orchestra.run.smithi064.stdout:4783 trunc from 0x71f6927 to 0x3ffadd5 2024-04-24T00:36:41.989 INFO:teuthology.orchestra.run.smithi064.stdout:4784 writesame 0x7758cf0 thru 0x775f592 data_size 0x1(0x68a3 bytes) 2024-04-24T00:36:41.995 INFO:teuthology.orchestra.run.smithi064.stdout:4786 trunc from 0x775f593 to 0x32f4b9f 2024-04-24T00:36:41.999 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:41.995+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:42.014 INFO:teuthology.orchestra.run.smithi064.stdout:4787 write 0x2188075 thru 0x219197b (0x9907 bytes) 2024-04-24T00:36:42.020 INFO:teuthology.orchestra.run.smithi064.stdout:4790 read 0xada802 thru 0xade034 (0x3833 bytes) 2024-04-24T00:36:42.048 INFO:teuthology.orchestra.run.smithi064.stdout:4792 write 0xc42546f thru 0xc426a76 (0x1608 bytes) 2024-04-24T00:36:42.055 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.055 INFO:teuthology.orchestra.run.smithi064.stdout:4796 writesame 0xc278c0b thru 0xc27cd80 data_size 0x3f(0x4176 bytes) 2024-04-24T00:36:42.062 INFO:teuthology.orchestra.run.smithi064.stdout:4797 write 0xbe0ba87 thru 0xbe193f1 (0xd96b bytes) 2024-04-24T00:36:42.076 INFO:teuthology.orchestra.run.smithi064.stdout:4798 punch from 0x3f14e5 to 0x3f7b89, (0x66a4 bytes) 2024-04-24T00:36:42.081 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:42.081 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:42.089 INFO:teuthology.orchestra.run.smithi064.stdout:4799 trunc from 0xc426a77 to 0x38b222e 2024-04-24T00:36:42.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:42.104 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:42.099+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:42.120 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.155 INFO:teuthology.orchestra.run.smithi064.stdout:4801 writesame 0xe1e5e52 thru 0xe1f2d95 data_size 0x23(0xcf44 bytes) 2024-04-24T00:36:42.155 INFO:teuthology.orchestra.run.smithi064.stdout:4803 read 0x3947fde thru 0x3953f2a (0xbf4d bytes) 2024-04-24T00:36:42.155 INFO:teuthology.orchestra.run.smithi064.stdout:4804 writesame 0x1760c0f thru 0x1766289 data_size 0xd(0x567b bytes) 2024-04-24T00:36:42.167 INFO:teuthology.orchestra.run.smithi064.stdout:4808 read 0x350b7b0 thru 0x35109c3 (0x5214 bytes) 2024-04-24T00:36:42.167 INFO:teuthology.orchestra.run.smithi064.stdout:4811 write 0x3e2019d thru 0x3e2fdde (0xfc42 bytes) 2024-04-24T00:36:42.182 INFO:teuthology.orchestra.run.smithi064.stdout:4813 write 0x28ce07e thru 0x28ce0d9 (0x5c bytes) 2024-04-24T00:36:42.186 INFO:teuthology.orchestra.run.smithi064.stdout:4814 writesame 0x3753855 thru 0x37605ed data_size 0x7(0xcd99 bytes) 2024-04-24T00:36:42.198 INFO:teuthology.orchestra.run.smithi064.stdout:4815 read 0xd8d37e6 thru 0xd8e25f0 (0xee0b bytes) 2024-04-24T00:36:42.198 INFO:teuthology.orchestra.run.smithi064.stdout:4816 read 0xd3b6c46 thru 0xd3c5be8 (0xefa3 bytes) 2024-04-24T00:36:42.198 INFO:teuthology.orchestra.run.smithi064.stdout:4818 write 0xc5bfa36 thru 0xc5cf849 (0xfe14 bytes) 2024-04-24T00:36:42.204 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:42.199+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:42.233 INFO:teuthology.orchestra.run.smithi064.stdout:4819 trunc from 0xe1f2d96 to 0x583e443 2024-04-24T00:36:42.249 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:42.249 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:42.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:42.283 INFO:teuthology.orchestra.run.smithi064.stdout:4820 write 0x7c1b7eb thru 0x7c228c1 (0x70d7 bytes) 2024-04-24T00:36:42.294 INFO:teuthology.orchestra.run.smithi064.stdout:4821 read 0x3aec93b thru 0x3af9e38 (0xd4fe bytes) 2024-04-24T00:36:42.294 INFO:teuthology.orchestra.run.smithi064.stdout:4822 read 0x3fd2105 thru 0x3fd519f (0x309b bytes) 2024-04-24T00:36:42.295 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.304 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:42.299+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:42.312 INFO:teuthology.orchestra.run.smithi064.stdout:4824 write 0xaa23d3b thru 0xaa2b061 (0x7327 bytes) 2024-04-24T00:36:42.336 INFO:teuthology.orchestra.run.smithi064.stdout:4825 write 0xd32e53a thru 0xd337ec6 (0x998d bytes) 2024-04-24T00:36:42.343 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.343 INFO:teuthology.orchestra.run.smithi064.stdout:4827 read 0x76a3dc4 thru 0x76b1834 (0xda71 bytes) 2024-04-24T00:36:42.363 INFO:teuthology.orchestra.run.smithi064.stdout:4828 trunc from 0xd337ec7 to 0x1270358 2024-04-24T00:36:42.405 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:42.399+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:42.407 INFO:teuthology.orchestra.run.smithi064.stdout:4829 read 0xa3417b thru 0xa3fe1f (0xbca5 bytes) 2024-04-24T00:36:42.408 INFO:teuthology.orchestra.run.smithi064.stdout:4830 punch from 0xc74753 to 0xc74ad3, (0x380 bytes) 2024-04-24T00:36:42.408 INFO:teuthology.orchestra.run.smithi064.stdout:4832 writesame 0x2f17d3 thru 0x2fd194 data_size 0x3e(0xb9c2 bytes) 2024-04-24T00:36:42.413 INFO:teuthology.orchestra.run.smithi064.stdout:4833 read 0x125bcf1 thru 0x126a9b0 (0xecc0 bytes) 2024-04-24T00:36:42.414 INFO:teuthology.orchestra.run.smithi064.stdout:4834 read 0xb64943 thru 0xb6da16 (0x90d4 bytes) 2024-04-24T00:36:42.416 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.416 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.416 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.416 INFO:teuthology.orchestra.run.smithi064.stdout:4838 read 0xa39074 thru 0xa44ad0 (0xba5d bytes) 2024-04-24T00:36:42.422 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:42.423 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:42.433 INFO:teuthology.orchestra.run.smithi064.stdout:4839 trunc from 0x1270358 to 0x968fb1e 2024-04-24T00:36:42.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:42.448 INFO:teuthology.orchestra.run.smithi064.stdout:4840 writesame 0x36958af thru 0x3697771 data_size 0x19(0x1ec3 bytes) 2024-04-24T00:36:42.456 INFO:teuthology.orchestra.run.smithi064.stdout:4841 punch from 0x6f4b52 to 0x6f4d96, (0x244 bytes) 2024-04-24T00:36:42.472 INFO:teuthology.orchestra.run.smithi064.stdout:4842 writesame 0x9def34f thru 0x9df7c78 data_size 0x2(0x892a bytes) 2024-04-24T00:36:42.476 INFO:teuthology.orchestra.run.smithi064.stdout:4843 trunc from 0x9df7c79 to 0xc042586 2024-04-24T00:36:42.492 INFO:teuthology.orchestra.run.smithi064.stdout:4844 writesame 0x13a0c9d thru 0x13aa169 data_size 0x1(0x94cd bytes) 2024-04-24T00:36:42.506 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:42.503+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:42.569 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:42.569 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:42.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:42.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:36:42.607 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:42.603+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:42.630 INFO:teuthology.orchestra.run.smithi064.stdout:4845 read 0xb16eb38 thru 0xb177b02 (0x8fcb bytes) 2024-04-24T00:36:42.630 INFO:teuthology.orchestra.run.smithi064.stdout:4846 writesame 0x916b5cf thru 0x916e6c1 data_size 0x1(0x30f3 bytes) 2024-04-24T00:36:42.664 INFO:teuthology.orchestra.run.smithi064.stdout:4847 read 0x9a52463 thru 0x9a53189 (0xd27 bytes) 2024-04-24T00:36:42.664 INFO:teuthology.orchestra.run.smithi064.stdout:4849 read 0xa19c646 thru 0xa19eb8c (0x2547 bytes) 2024-04-24T00:36:42.664 INFO:teuthology.orchestra.run.smithi064.stdout:4850 write 0x79eb832 thru 0x79f0ac0 (0x528f bytes) 2024-04-24T00:36:42.671 INFO:teuthology.orchestra.run.smithi064.stdout:4851 write 0xab6a243 thru 0xab77144 (0xcf02 bytes) 2024-04-24T00:36:42.698 INFO:teuthology.orchestra.run.smithi064.stdout:4853 write 0xda16ef8 thru 0xda19653 (0x275c bytes) 2024-04-24T00:36:42.703 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.703 INFO:teuthology.orchestra.run.smithi064.stdout:4856 write 0x3bbaabd thru 0x3bbcaaa (0x1fee bytes) 2024-04-24T00:36:42.708 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:42.703+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:42.710 INFO:teuthology.orchestra.run.smithi064.stdout:4857 punch from 0xb49f9d4 to 0xb4a3284, (0x38b0 bytes) 2024-04-24T00:36:42.710 INFO:teuthology.orchestra.run.smithi064.stdout:4858 read 0x95641bb thru 0x95704ce (0xc314 bytes) 2024-04-24T00:36:42.710 INFO:teuthology.orchestra.run.smithi064.stdout:4859 read 0xd5b5c90 thru 0xd5bad6c (0x50dd bytes) 2024-04-24T00:36:42.710 INFO:teuthology.orchestra.run.smithi064.stdout:4860 punch from 0x4d141fa to 0x4d22989, (0xe78f bytes) 2024-04-24T00:36:42.711 INFO:teuthology.orchestra.run.smithi064.stdout:4862 punch from 0xc4d4613 to 0xc4d82dd, (0x3cca bytes) 2024-04-24T00:36:42.711 INFO:teuthology.orchestra.run.smithi064.stdout:4864 writesame 0x5454e09 thru 0x54599d0 data_size 0x32(0x4bc8 bytes) 2024-04-24T00:36:42.718 INFO:teuthology.orchestra.run.smithi064.stdout:4865 trunc from 0xda19654 to 0xdb27090 2024-04-24T00:36:42.733 INFO:teuthology.orchestra.run.smithi064.stdout:4866 write 0x70901c3 thru 0x709978e (0x95cc bytes) 2024-04-24T00:36:42.745 INFO:teuthology.orchestra.run.smithi064.stdout:4868 write 0x664f20b thru 0x6656945 (0x773b bytes) 2024-04-24T00:36:42.750 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.750 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:42.755 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:42.755 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:42.765 INFO:teuthology.orchestra.run.smithi064.stdout:4873 trunc from 0xdb27090 to 0x3635d0f 2024-04-24T00:36:42.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:42.808 INFO:teuthology.orchestra.run.smithi064.stdout:4874 write 0xa65797e thru 0xa658597 (0xc1a bytes) 2024-04-24T00:36:42.810 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:42.807+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:42.813 INFO:teuthology.orchestra.run.smithi064.stdout:4875 read 0x78a1a47 thru 0x78a63e0 (0x499a bytes) 2024-04-24T00:36:42.813 INFO:teuthology.orchestra.run.smithi064.stdout:4876 read 0x779d0b7 thru 0x77ac510 (0xf45a bytes) 2024-04-24T00:36:42.813 INFO:teuthology.orchestra.run.smithi064.stdout:4877 punch from 0x946932d to 0x9477637, (0xe30a bytes) 2024-04-24T00:36:42.814 INFO:teuthology.orchestra.run.smithi064.stdout:4878 trunc from 0xa658598 to 0x9348028 2024-04-24T00:36:42.829 INFO:teuthology.orchestra.run.smithi064.stdout:4879 write 0x77839d9 thru 0x7784bc4 (0x11ec bytes) 2024-04-24T00:36:42.838 INFO:teuthology.orchestra.run.smithi064.stdout:4880 trunc from 0x9348028 to 0xad0f0a3 2024-04-24T00:36:42.850 INFO:teuthology.orchestra.run.smithi064.stdout:4881 punch from 0x941a69b to 0x94212ef, (0x6c54 bytes) 2024-04-24T00:36:42.850 INFO:teuthology.orchestra.run.smithi064.stdout:4883 writesame 0x4f9350f thru 0x4f95e40 data_size 0x1(0x2932 bytes) 2024-04-24T00:36:42.867 INFO:teuthology.orchestra.run.smithi064.stdout:4885 writesame 0x74527d2 thru 0x7453fc0 data_size 0x1(0x17ef bytes) 2024-04-24T00:36:42.879 INFO:teuthology.orchestra.run.smithi064.stdout:4886 punch from 0x56043f7 to 0x560b2b3, (0x6ebc bytes) 2024-04-24T00:36:42.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:42.901 INFO:teuthology.orchestra.run.smithi064.stdout:4888 write 0xe85d0e1 thru 0xe868611 (0xb531 bytes) 2024-04-24T00:36:42.910 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:42.907+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:42.911 INFO:teuthology.orchestra.run.smithi064.stdout:4889 read 0x28a51ac thru 0x28b0e07 (0xbc5c bytes) 2024-04-24T00:36:42.911 INFO:teuthology.orchestra.run.smithi064.stdout:4892 read 0x7399ef2 thru 0x73a6c5a (0xcd69 bytes) 2024-04-24T00:36:42.911 INFO:teuthology.orchestra.run.smithi064.stdout:4893 read 0x67678d thru 0x67c5e7 (0x5e5b bytes) 2024-04-24T00:36:42.912 INFO:teuthology.orchestra.run.smithi064.stdout:4894 read 0x70c5162 thru 0x70c6522 (0x13c1 bytes) 2024-04-24T00:36:42.913 INFO:teuthology.orchestra.run.smithi064.stdout:4896 write 0x2298f34 thru 0x229c010 (0x30dd bytes) 2024-04-24T00:36:42.917 INFO:teuthology.orchestra.run.smithi064.stdout:4897 write 0x6b64972 thru 0x6b6c7bc (0x7e4b bytes) 2024-04-24T00:36:42.926 INFO:teuthology.orchestra.run.smithi064.stdout:4898 read 0x6cf559f thru 0x6cf5724 (0x186 bytes) 2024-04-24T00:36:42.926 INFO:teuthology.orchestra.run.smithi064.stdout:4899 read 0xe13fe54 thru 0xe1407fb (0x9a8 bytes) 2024-04-24T00:36:42.930 INFO:teuthology.orchestra.run.smithi064.stdout:4903 trunc from 0xe868612 to 0xbf44071 2024-04-24T00:36:42.931 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:42.931 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:42.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:42.954 INFO:teuthology.orchestra.run.smithi064.stdout:4905 write 0x65a607 thru 0x65db35 (0x352f bytes) 2024-04-24T00:36:42.958 INFO:teuthology.orchestra.run.smithi064.stdout:4907 writesame 0x3929044 thru 0x3936db6 data_size 0x3(0xdd73 bytes) 2024-04-24T00:36:42.992 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:36:43.011 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:43.007+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:43.048 INFO:teuthology.orchestra.run.smithi064.stdout:truncating to largest ever: 0xede7d16 2024-04-24T00:36:43.052 INFO:teuthology.orchestra.run.smithi064.stdout:4909 trunc from 0xbf44071 to 0xede7d16 2024-04-24T00:36:43.067 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:43.067 INFO:teuthology.orchestra.run.smithi064.stdout:4914 write 0x6f29f8d thru 0x6f34c6c (0xace0 bytes) 2024-04-24T00:36:43.074 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:43.074 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:43.082 INFO:teuthology.orchestra.run.smithi064.stdout:4915 writesame 0xb00c410 thru 0xb01bf37 data_size 0x39(0xfb28 bytes) 2024-04-24T00:36:43.089 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:43.093 INFO:teuthology.orchestra.run.smithi064.stdout:4917 write 0xc63f12b thru 0xc6482fa (0x91d0 bytes) 2024-04-24T00:36:43.105 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:43.105 INFO:teuthology.orchestra.run.smithi064.stdout:4919 write 0xca80170 thru 0xca8dff8 (0xde89 bytes) 2024-04-24T00:36:43.111 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:43.107+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:43.115 INFO:teuthology.orchestra.run.smithi064.stdout:4920 punch from 0x29cd8be to 0x29d6a53, (0x9195 bytes) 2024-04-24T00:36:43.115 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:43.115 INFO:teuthology.orchestra.run.smithi064.stdout:4923 punch from 0x82c0f63 to 0x82c9a1a, (0x8ab7 bytes) 2024-04-24T00:36:43.115 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:43.135 INFO:teuthology.orchestra.run.smithi064.stdout:4925 trunc from 0xede7d16 to 0x24d7589 2024-04-24T00:36:43.184 INFO:teuthology.orchestra.run.smithi064.stdout:4926 write 0x4bdde97 thru 0x4be5454 (0x75be bytes) 2024-04-24T00:36:43.194 INFO:teuthology.orchestra.run.smithi064.stdout:4929 trunc from 0x4be5455 to 0x3d90c7a 2024-04-24T00:36:43.213 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:43.207+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:43.218 INFO:teuthology.orchestra.run.smithi064.stdout:4931 punch from 0x2a91488 to 0x2aa0139, (0xecb1 bytes) 2024-04-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:43.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:36:43.227+0000 7f468bfff640 0 --2- 172.21.15.64:0/763749836 >> v2:172.21.15.64:3300/0 conn(0x7f468c14a1b0 0x7f468c14c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-24T00:36:43.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:43.241 INFO:teuthology.orchestra.run.smithi064.stdout:4932 writesame 0x9913388 thru 0x9920219 data_size 0x89(0xce92 bytes) 2024-04-24T00:36:43.241 INFO:teuthology.orchestra.run.smithi064.stdout:4933 punch from 0x8ff6b57 to 0x9000f7a, (0xa423 bytes) 2024-04-24T00:36:43.257 INFO:teuthology.orchestra.run.smithi064.stdout:4937 writesame 0xc112e2d thru 0xc116594 data_size 0x24(0x3768 bytes) 2024-04-24T00:36:43.257 INFO:teuthology.orchestra.run.smithi064.stdout:4938 write 0xa13cc80 thru 0xa1435db (0x695c bytes) 2024-04-24T00:36:43.259 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:43.260 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:43.265 INFO:teuthology.orchestra.run.smithi064.stdout:4939 read 0x3c8d0fb thru 0x3c99931 (0xc837 bytes) 2024-04-24T00:36:43.265 INFO:teuthology.orchestra.run.smithi064.stdout:4941 write 0x465a111 thru 0x4664211 (0xa101 bytes) 2024-04-24T00:36:43.274 INFO:teuthology.orchestra.run.smithi064.stdout:4942 read 0x1e4fdff thru 0x1e5cc5d (0xce5f bytes) 2024-04-24T00:36:43.278 INFO:teuthology.orchestra.run.smithi064.stdout:4947 trunc from 0xc116595 to 0x9bb6a97 2024-04-24T00:36:43.283 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:43.283 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:43.293 INFO:teuthology.orchestra.run.smithi064.stdout:4949 read 0x4f1dc42 thru 0x4f28e48 (0xb207 bytes) 2024-04-24T00:36:43.295 INFO:teuthology.orchestra.run.smithi064.stdout:4950 trunc from 0x9bb6a97 to 0xad8cf10 2024-04-24T00:36:43.300 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:43.301 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:43.308 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:43.308 INFO:teuthology.orchestra.run.smithi064.stdout:4952 writesame 0x893dcf3 thru 0x8949e11 data_size 0x1(0xc11f bytes) 2024-04-24T00:36:43.313 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:43.307+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:43.318 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:43.318 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:43.339 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:43.340 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:43.361 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:43.362 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:43.382 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.17188858599999, 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.17153767100001, 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.16464935100001, 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.906367724999996, 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:43.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:43.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:43.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:43.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.896091308999999, 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:43.413 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.434140506000006, 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:43.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.421732239999997, 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:43.423 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.421700486000006, 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:43.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:43.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:43.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.418167990000001, 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.428 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:43.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:43.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.408106485000005, 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:43.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:43.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.194163578000001, 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:43.436 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.184151634000003, 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:43.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.910772772000001, 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:43.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:43.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:43.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.894104323000001, 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:43.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:43.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.708206539999999, 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:43.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:43.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.024448182999997, 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:43.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.002231184999999, 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:43.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:43.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:43.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.841009905, 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:43.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:43.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:43.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.697661347, 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.685036387, 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:43.481 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:43.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:43.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:43.485 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:43.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:43.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:43.488 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:43.489 INFO:teuthology.orchestra.run.smithi064.stdout:4953 read 0x5d1b48f thru 0x5d24963 (0x94d5 bytes) 2024-04-24T00:36:43.489 INFO:teuthology.orchestra.run.smithi064.stdout:4954 read 0x1c1691 thru 0x1c43c3 (0x2d33 bytes) 2024-04-24T00:36:43.489 INFO:teuthology.orchestra.run.smithi064.stdout:4956 write 0x2d541c thru 0x2e4014 (0xebf9 bytes) 2024-04-24T00:36:43.489 INFO:teuthology.orchestra.run.smithi064.stdout:4958 trunc from 0xad8cf10 to 0xe6b61b5 2024-04-24T00:36:43.489 INFO:teuthology.orchestra.run.smithi064.stdout:4959 punch from 0x2392ccc to 0x2395689, (0x29bd bytes) 2024-04-24T00:36:43.489 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:43.489 INFO:teuthology.orchestra.run.smithi064.stdout:4961 punch from 0x9a2898d to 0x9a3212a, (0x979d bytes) 2024-04-24T00:36:43.490 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:43.483+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:43.501 INFO:teuthology.orchestra.run.smithi064.stdout:4962 trunc from 0xe6b61b5 to 0x1f3858b 2024-04-24T00:36:43.527 INFO:teuthology.orchestra.run.smithi064.stdout:4963 punch from 0xde9146 to 0xded1a7, (0x4061 bytes) 2024-04-24T00:36:43.527 INFO:teuthology.orchestra.run.smithi064.stdout:4966 punch from 0x1a771a8 to 0x1a7ef8d, (0x7de5 bytes) 2024-04-24T00:36:43.551 INFO:teuthology.orchestra.run.smithi064.stdout:4967 writesame 0x78559ab thru 0x7860e55 data_size 0x1b(0xb4ab bytes) 2024-04-24T00:36:43.566 INFO:teuthology.orchestra.run.smithi064.stdout:4968 write 0x9ae01e3 thru 0x9ae4e04 (0x4c22 bytes) 2024-04-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:43.588 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:43.583+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:43.588 INFO:teuthology.orchestra.run.smithi064.stdout:4970 write 0xb083055 thru 0xb0845b6 (0x1562 bytes) 2024-04-24T00:36:43.607 INFO:teuthology.orchestra.run.smithi064.stdout:4971 trunc from 0xb0845b7 to 0x60fee1f 2024-04-24T00:36:43.624 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:43.624 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:43.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.624 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:125936 3.9 3:93d47636:::rbd_object_map.115ee44a9909:head [call lock.assert_locked in=44b,call rbd.object_map_resize in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-04-24T00:36:43.624 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:43.623088+0000", 2024-04-24T00:36:43.624 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00046804799999999999, 2024-04-24T00:36:43.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00050158400000000001, 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 125936 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.625 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623088+0000", 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623088+0000", 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623090+0000", 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.973e-06 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623100+0000", 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0134e-05 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623125+0000", 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4842999999999999e-05 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623144+0000", 2024-04-24T00:36:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9247000000000001e-05 2024-04-24T00:36:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623227+0000", 2024-04-24T00:36:43.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2812000000000006e-05 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623291+0000", 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3941000000000003e-05 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:43.623488+0000", 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000196889 2024-04-24T00:36:43.629 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.630 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.630 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.630 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.630 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:36:43.630 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:36:43.630 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:43.633 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:43.633 INFO:teuthology.orchestra.run.smithi064.stdout:4973 clone 34 order 19 su 2048 sc 2 2024-04-24T00:36:43.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:43.689 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:43.683+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:43.762 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:43.762 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:43.762 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:43.762 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:43.762 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:43.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:43.822 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:43.819+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:43.891 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:43.887+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:43.944 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:43.944 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:43.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:43.944 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:43.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.72541990799999, 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:43.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:43.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:43.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:43.950 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.303634214, 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:43.955 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.161506648, 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:43.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:43.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:43.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 70.023734865999998, 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.962 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:43.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:43.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.967 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.986252547999996, 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.968 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.969 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:43.970 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.971 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.972 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.973 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81770 3.e 3:70ed8754:::rbd_data.115e6f07caf6.0000000000000008:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.956923586000002, 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069468495000000005, 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.974 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81770 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982821+0000", 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.975 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.982823+0000", 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0889999999999998e-06 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987146+0000", 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00432307 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.976 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987181+0000", 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4932999999999999e-05 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.987199+0000", 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7723999999999998e-05 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998119+0000", 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010919936 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.977 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998250+0000", 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013053699999999999 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.998402+0000", 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015274099999999999 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027487+0000", 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029084648000000001 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.978 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052191+0000", 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02470412 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052254+0000", 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3330000000000005e-05 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.052290+0000", 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5367000000000002e-05 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.979 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.759661683000004, 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:43.980 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.981 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.982 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.323906683999994, 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:43.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:43.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:43.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:43.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:43.991 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.314048647999996, 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:43.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:43.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:43.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:43.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:43.997 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 66.293699040000007, 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:43.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:43.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.003 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.259653704999998, 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:44.007 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.257343869000003, 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.008 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:44.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:44.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.248775070000001, 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.014 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:44.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:44.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.309625977000003, 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.020 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:44.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:44.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.420235980999998, 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:44.026 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:44.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:44.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.376760183999998, 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.032 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:44.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.402786882000001, 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.036 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:44.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:44.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:44.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.383644979000003, 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:44.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:44.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:44.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.424357426, 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.046 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:44.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:44.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.051 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.390355004, 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:44.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.055 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.056 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.056 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:44.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:44.058 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:44.055+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:44.158 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:44.155+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:44.187 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:44.187 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:44.187 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:44.187 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:44.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:44.289 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:44.287+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:44.362 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:44.355+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:44.367 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:44.367 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:44.367 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:44.367 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:44.368 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:44.381 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:44.469 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:44.467+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:44.552 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.83994822099999, 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:44.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:44.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:44.558 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.83985029600001, 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.559 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:44.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:36:44.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:36:44.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.480470764, 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.566 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:44.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:36:44.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:44659 3.1d 3:bb1ec1ca:::rbd_data.115ec7ed39de.0000000000000003:head [write 0~393216 [fadvise_sequential+fadvise_nocache],write 655360~60371 [fadvise_sequential+fadvise_nocache],write 1179648~1127107 [fadvise_sequential+fadvise_nocache],write 10622976~4096 [fadvise_sequential+fadvise_nocache],write 10635264~4096 [fadvise_sequential+fadvise_nocache],write 10647552~4096 [fadvise_sequential+fadvise_nocache],write 10659840~4096 [fadvise_sequential+fadvise_nocache],write 10672128~4096 [fadvise_sequential+fadvise_nocache],write 10684416~4096 [fadvise_sequential+fadvise_nocache],write 10696704~4096 [fadvise_sequential+fadvise_nocache],write 10708992~4096 [fadvise_sequential+fadvise_nocache],write 10721280~4096 [fadvise_sequential+fadvise_nocache],write 10733568~4096 [fadvise_sequential+fadvise_nocache],write 10745856~2048 [fadvise_sequential+fadvise_nocache],write 10749952~4096 [fadvise_sequential+fadvise_nocache],write 10762240~4096 [fadvise_sequential+fadvise_nocache],write 10774528~4096 [fadvise_sequential+fadvise_nocache],write 10786816~4096 [fadvise_sequential+fadvise_nocache],write 10799104~4096 [fadvise_sequential+fadvise_nocache],write 10811392~4096 [fadvise_sequential+fadvise_nocache],write 10823680~4096 [fadvise_sequential+fadvise_nocache],write 10835968~4096 [fadvise_sequential+fadvise_nocache],write 10848256~4096 [fadvise_sequential+fadvise_nocache],write 10860544~4096 [fadvise_sequential+fadvise_nocache],write 10872832~4096 [fadvise_sequential+fadvise_nocache],write 10878976~2048 [fadvise_sequential+fadvise_nocache],write 10889216~4096 [fadvise_sequential+fadvise_nocache],write 10901504~4096 [fadvise_sequential+fadvise_nocache],write 10913792~4096 [fadvise_sequential+fadvise_nocache],write 10926080~4096 [fadvise_sequential+fadvise_nocache],write 10938368~4096 [fadvise_sequential+fadvise_nocache],write 10950656~4096 [fadvise_sequential+fadvise_nocache],write 10962944~4096 [fadvise_sequential+fadvise_nocache],write 10975232~4096 [fadvise_sequential+fadvise_nocache],write 10987520~4096 [fadvise_sequential+fadvise_nocache],write 10999808~4096 [fadvise_sequential+fadvise_nocache],write 11016192~4096 [fadvise_sequential+fadvise_nocache],write 11028480~4096 [fadvise_sequential+fadvise_nocache],write 11040768~4096 [fadvise_sequential+fadvise_nocache],write 11053056~4096 [fadvise_sequential+fadvise_nocache],write 11065344~4096 [fadvise_sequential+fadvise_nocache],write 11077632~4096 [fadvise_sequential+fadvise_nocache],write 11089920~4096 [fadvise_sequential+fadvise_nocache],write 11102208~4096 [fadvise_sequential+fadvise_nocache],write 11114496~4096 [fadvise_sequential+fadvise_nocache],write 11126784~4096 [fadvise_sequential+fadvise_nocache],write 11139072~2048 [fadvise_sequential+fadvise_nocache],write 11143168~3210 [fadvise_sequential+fadvise_nocache],write 14422016~8192 [fadvise_sequential+fadvise_nocache],write 14434304~8192 [fadvise_sequential+fadvise_nocache],write 14446592~8192 [fadvise_sequential+fadvise_nocache],write 14458880~8192 [fadvise_sequential+fadvise_nocache],write 14471168~8192 [fadvise_sequential+fadvise_nocache],write 14483456~8192 [fadvise_sequential+fadvise_nocache],write 14495744~8192 [fadvise_sequential+fadvise_nocache],write 14508032~8192 [fadvise_sequential+fadvise_nocache],write 14520320~8192 [fadvise_sequential+fadvise_nocache],write 14532608~8192 [fadvise_sequential+fadvise_nocache],write 14544896~12288 [fadvise_sequential+fadvise_nocache],write 14561280~8192 [fadvise_sequential+fadvise_nocache],write 14573568~8192 [fadvise_sequential+fadvise_nocache],write 14585856~8192 [fadvise_sequential+fadvise_nocache],write 14598144~8192 [fadvise_sequential+fadvise_nocache],write 14610432~8192 [fadvise_sequential+fadvise_nocache],write 14622720~8192 [fadvise_sequential+fadvise_nocache],write 14635008~8192 [fadvise_sequential+fadvise_nocache],write 14647296~8192 [fadvise_sequential+fadvise_nocache],write 14659584~8192 [fadvise_sequential+fadvise_nocache],write 14671872~8192 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 107.657486777, 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.066878900000000005, 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.572 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44659 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890829+0000", 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.890831+0000", 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.911e-06 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894074+0000", 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032425029999999999 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894096+0000", 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2127999999999999e-05 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894108+0000", 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1943999999999999e-05 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894200+0000", 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1700999999999995e-05 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.894589+0000", 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000388988 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.895654+0000", 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010655759999999999 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.934339+0000", 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038685086 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957400+0000", 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023061102 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957596+0000", 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019601199999999999 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:56.957708+0000", 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011194899999999999 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.577 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.06972663800001, 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:36:44.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:36:44.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.583 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.04445803199999, 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:36:44.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:36:44.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:36:44.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:36:44.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:36:44.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:36:44.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:36:44.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.191254311999998, 2024-04-24T00:36:44.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:36:44.590 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.590 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.590 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.590 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.590 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.590 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.590 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:36:44.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.591 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:44.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:36:44.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:36:44.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.596 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.158649198000006, 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:44.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.600 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.601 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.601 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:44.601 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:44.601 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 85.754074504000002, 2024-04-24T00:36:44.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:36:44.601 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.602 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.602 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.602 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.602 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.602 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.602 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:36:44.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.603 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:44.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:44.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:36:44.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:36:44.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:36:44.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:36:44.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:36:44.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:36:44.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:36:44.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:36:44.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:36:44.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:36:44.610 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.611 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.611 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.611 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:36:44.611 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:44.611 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 85.696234274000005, 2024-04-24T00:36:44.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:36:44.612 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.612 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.612 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.612 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:44.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:44.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:36:44.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:36:44.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:36:44.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:36:44.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:36:44.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:36:44.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:36:44.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:36:44.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:36:44.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:36:44.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:36:44.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:36:44.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:36:44.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:36:44.620 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.620 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.620 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.620 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89931 3.1f 3:fff09b5a:::rbd_data.115e56aa02fc.0000000000000003:head [write 0~4577672 [fadvise_sequential+fadvise_nocache],zero 4718592~262144,zero 6291456~179747,write 6471203~13759 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:44.621 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:44.621 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.856701671000003, 2024-04-24T00:36:44.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.068626284999999995, 2024-04-24T00:36:44.621 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.621 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.621 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.622 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.622 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.622 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.622 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89931 2024-04-24T00:36:44.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.622 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:44.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691614+0000", 2024-04-24T00:36:44.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691618+0000", 2024-04-24T00:36:44.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2949999999999998e-06 2024-04-24T00:36:44.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704054+0000", 2024-04-24T00:36:44.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012436665 2024-04-24T00:36:44.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704090+0000", 2024-04-24T00:36:44.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5778999999999997e-05 2024-04-24T00:36:44.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704107+0000", 2024-04-24T00:36:44.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7122999999999999e-05 2024-04-24T00:36:44.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704255+0000", 2024-04-24T00:36:44.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014735099999999999 2024-04-24T00:36:44.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704588+0000", 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033309499999999999 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.705598+0000", 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001010597 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.758968+0000", 2024-04-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053369708000000002 2024-04-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760090+0000", 2024-04-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011214840000000001 2024-04-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760173+0000", 2024-04-24T00:36:44.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3147999999999997e-05 2024-04-24T00:36:44.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.760241+0000", 2024-04-24T00:36:44.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8040000000000006e-05 2024-04-24T00:36:44.632 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.632 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.632 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.632 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:36:44.632 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 55.854671924999998, 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:44.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:44.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:36:44.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:44.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:36:44.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90259 3.2 3:4377c456:::rbd_data.115e56aa02fc.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.018188760000001, 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.064637350999999996, 2024-04-24T00:36:44.638 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90259 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530127+0000", 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530129+0000", 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9640000000000002e-06 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530137+0000", 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:36:44.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530152+0000", 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4612e-05 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530160+0000", 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-06 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530341+0000", 2024-04-24T00:36:44.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018113600000000001 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.530535+0000", 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193606 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.534329+0000", 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037947839999999998 2024-04-24T00:36:44.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594447+0000", 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060117469999999999 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594580+0000", 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013329699999999999 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594679+0000", 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9085000000000004e-05 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.594765+0000", 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5533999999999998e-05 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:44.644 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.875924374, 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:36:44.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:36:44.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:36:44.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:36:44.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:36:44.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:36:44.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:36:44.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:36:44.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.651 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.652 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:36:44.652 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:44.652 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.846282320999997, 2024-04-24T00:36:44.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:36:44.652 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.653 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.653 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.653 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:44.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:44.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:36:44.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:44.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:36:44.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:36:44.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:36:44.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:36:44.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:36:44.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:36:44.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:36:44.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:36:44.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:36:44.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:36:44.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:36:44.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:36:44.659 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.659 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.659 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.660 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:36:44.660 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:44.660 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.699766910000001, 2024-04-24T00:36:44.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:36:44.661 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.661 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.661 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.661 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.661 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.661 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:36:44.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.910994352000003, 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:36:44.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:44.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:36:44.673 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.887850939000003, 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:36:44.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:36:44.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.014542191, 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:36:44.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:36:44.683 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.961859914000001, 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:44.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:44.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:44.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.688 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:44.688 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:44.688 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:44.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:44.690 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:44.687+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:44.790 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:44.787+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:44.890 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:44.887+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:44.899 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:44.899 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:44.899 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:44.899 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:44.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:45.005 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:45.003+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:45.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:36:45.093 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:45.091+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:45.148 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:45.148 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:45.148 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:45.149 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:45.149 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:45.172 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:45.215 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:45.211+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:45.219 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5, 0] out_osds: [7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:45.219 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:45.219 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-24T00:36:45.219 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:36:45.220 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:36:45.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-04-24T00:36:45.301 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:45.299+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:45.397 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:45.391+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:45.406 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:45.406 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.87148953100001, 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.407 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:36:45.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:36:45.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.83745247799999, 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:36:45.413 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:36:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.722338482, 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:36:45.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:36:45.423 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.694888521, 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:36:45.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:36:45.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:36:45.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.774816689000005, 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:36:45.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:36:45.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.768545431000007, 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:45.433 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:36:45.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:36:45.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.757090844999993, 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:36:45.439 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:36:45.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:36:45.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 67.702638300999993, 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.444 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.448 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105171 3.1e 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head [zero 0~262144,write 262144~131072 [fadvise_sequential+fadvise_nocache],zero 393216~131072,write 524288~917504 [fadvise_sequential+fadvise_nocache],zero 1441792~393216,write 1835008~131072 [fadvise_sequential+fadvise_nocache],write 2097152~917504 [fadvise_sequential+fadvise_nocache],zero 3014656~393216,write 3407872~131072 [fadvise_sequential+fadvise_nocache],zero 3538944~131072,write 3670016~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.712887809000001, 2024-04-24T00:36:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061400402, 2024-04-24T00:36:45.449 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105171 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690858+0000", 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5059999999999998e-06 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697752+0000", 2024-04-24T00:36:45.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006893766 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697795+0000", 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2589000000000002e-05 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697810+0000", 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5620999999999998e-05 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697936+0000", 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012529199999999999 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698150+0000", 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021430200000000001 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698445+0000", 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029468099999999998 2024-04-24T00:36:45.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736712+0000", 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038267781000000001 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752153+0000", 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015441043999999999 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752204+0000", 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752256+0000", 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2286e-05 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127047 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 374784~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650011+0000", 2024-04-24T00:36:45.455 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.75373276300000003, 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055335291000000002, 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127047 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650011+0000", 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650011+0000", 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650012+0000", 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650016+0000", 2024-04-24T00:36:45.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1980000000000003e-06 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650028+0000", 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.169e-05 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650036+0000", 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7360000000000007e-06 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.704970+0000", 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054933370000000002 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705049+0000", 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9080999999999997e-05 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705346+0000", 2024-04-24T00:36:45.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029734199999999997 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127048 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 378880~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.75369516299999995, 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768986, 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.460 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127048 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:45.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.92e-07 2024-04-24T00:36:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650053+0000", 2024-04-24T00:36:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.698e-06 2024-04-24T00:36:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650065+0000", 2024-04-24T00:36:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1454000000000001e-05 2024-04-24T00:36:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650073+0000", 2024-04-24T00:36:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8920000000000006e-06 2024-04-24T00:36:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705378+0000", 2024-04-24T00:36:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055304820999999997 2024-04-24T00:36:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705456+0000", 2024-04-24T00:36:45.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8009999999999993e-05 2024-04-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705818+0000", 2024-04-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036131899999999997 2024-04-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127049 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 382976~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.469 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.75365799099999997, 2024-04-24T00:36:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056235535000000003, 2024-04-24T00:36:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.470 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127049 2024-04-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:36:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650090+0000", 2024-04-24T00:36:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:36:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650101+0000", 2024-04-24T00:36:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0822000000000001e-05 2024-04-24T00:36:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650110+0000", 2024-04-24T00:36:45.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-04-24T00:36:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705851+0000", 2024-04-24T00:36:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055741377000000002 2024-04-24T00:36:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705916+0000", 2024-04-24T00:36:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5160000000000006e-05 2024-04-24T00:36:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706321+0000", 2024-04-24T00:36:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040486500000000002 2024-04-24T00:36:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127050 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 387072~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.75362253000000001, 2024-04-24T00:36:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056659432000000003, 2024-04-24T00:36:45.480 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.481 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127050 2024-04-24T00:36:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:45.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:45.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650122+0000", 2024-04-24T00:36:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:36:45.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650125+0000", 2024-04-24T00:36:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.366e-06 2024-04-24T00:36:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650136+0000", 2024-04-24T00:36:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0624e-05 2024-04-24T00:36:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650144+0000", 2024-04-24T00:36:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4360000000000002e-06 2024-04-24T00:36:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706354+0000", 2024-04-24T00:36:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056210006999999999 2024-04-24T00:36:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706437+0000", 2024-04-24T00:36:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2515000000000005e-05 2024-04-24T00:36:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706781+0000", 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034373599999999998 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127051 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.75358857800000001, 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057055097999999999, 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.489 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127051 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650156+0000", 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1399999999999996e-07 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650159+0000", 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3100000000000001e-06 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650170+0000", 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.058e-05 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650178+0000", 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5660000000000007e-06 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706827+0000", 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056648506000000001 2024-04-24T00:36:45.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706903+0000", 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6493000000000001e-05 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.707210+0000", 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306929 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127334 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664445+0000", 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.73929896500000003, 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055399756000000001, 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127334 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.494 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664445+0000", 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664445+0000", 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664445+0000", 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9599999999999999e-07 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664448+0000", 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3240000000000001e-06 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664454+0000", 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5919999999999997e-06 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664463+0000", 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1290000000000002e-06 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719463+0000", 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054999271000000002 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719519+0000", 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6206999999999999e-05 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719845+0000", 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032573699999999999 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127335 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.73927296200000003, 2024-04-24T00:36:45.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055806472000000003, 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127335 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5200000000000002e-07 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664474+0000", 2024-04-24T00:36:45.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6400000000000001e-06 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664481+0000", 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.002e-06 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664493+0000", 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2311e-05 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719879+0000", 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055386187000000003 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719956+0000", 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6196e-05 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720277+0000", 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032168399999999998 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.502 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127336 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.73924045000000005, 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056212985, 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.503 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127336 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664504+0000", 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5400000000000001e-07 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664513+0000", 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.132e-06 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664520+0000", 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3849999999999996e-06 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664525+0000", 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0180000000000001e-06 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720371+0000", 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055845976999999998 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720430+0000", 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8964e-05 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720716+0000", 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028595500000000001 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127337 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.73920907700000005, 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056616423999999999, 2024-04-24T00:36:45.507 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127337 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2099999999999997e-07 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664538+0000", 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0410000000000002e-06 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664546+0000", 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3710000000000002e-06 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664551+0000", 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0889999999999999e-06 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720759+0000", 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056208658000000002 2024-04-24T00:36:45.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720811+0000", 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1564999999999998e-05 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721151+0000", 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034017900000000002 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127338 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.73918493699999999, 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057296623999999997, 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127338 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.58e-07 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664562+0000", 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.345e-06 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664568+0000", 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664573+0000", 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:36:45.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721197+0000", 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056624102000000003 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721296+0000", 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9001999999999998e-05 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721855+0000", 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055934300000000001 2024-04-24T00:36:45.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127339 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.73916439099999998, 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057818889999999998, 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:45.516 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127339 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:45.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664580+0000", 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5299999999999999e-07 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664582+0000", 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1399999999999998e-06 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664588+0000", 2024-04-24T00:36:45.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1959999999999996e-06 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664593+0000", 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.656e-06 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721894+0000", 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057301501999999997 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721991+0000", 2024-04-24T00:36:45.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6330999999999996e-05 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.722398+0000", 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040761200000000001 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:45.520 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:45.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:45.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:45.523 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:45.524 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "" 2024-04-24T00:36:45.524 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:45.524 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:45.519+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:45.623 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:45.619+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:45.697 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:45.698 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:45.724 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:45.725 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:45.719+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:45.725 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:45.745 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:45.745 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:45.745 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:45.745 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:45.746 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:45.746 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:45.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:45.767 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:45.767 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:45.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:45.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:45.815 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:45.815 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:45.835 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:45.835 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:45.853 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:45.851+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:45.860 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:45.861 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:45.926 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:45.923+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:45.932 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:45.933 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:45.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:46.016 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:36:46.027 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:46.023+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:46.108 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:46.108 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:46.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:46.129 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-04-24T00:36:46.129 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:46.127+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:46.229 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:46.227+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:46.271 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:46.271 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:46.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:46.330 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:46.327+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:46.431 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:46.427+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:46.463 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:46.463 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:46.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:46.503 INFO:teuthology.orchestra.run.smithi064.stdout:4974 read 0xa30e0b thru 0xa39491 (0x8687 bytes) 2024-04-24T00:36:46.505 INFO:teuthology.orchestra.run.smithi064.stdout:4975 trunc from 0x60fee1f to 0x67b083a 2024-04-24T00:36:46.519 INFO:teuthology.orchestra.run.smithi064.stdout:4976 punch from 0x9bf0e2 to 0x9c9c94, (0xabb2 bytes) 2024-04-24T00:36:46.531 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:46.527+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:46.539 INFO:teuthology.orchestra.run.smithi064.stdout:4977 writesame 0xd8d5922 thru 0xd8d8e03 data_size 0x7(0x34e2 bytes) 2024-04-24T00:36:46.539 INFO:teuthology.orchestra.run.smithi064.stdout:4978 punch from 0x20b1b10 to 0x20bad50, (0x9240 bytes) 2024-04-24T00:36:46.539 INFO:teuthology.orchestra.run.smithi064.stdout:4979 write 0xa195d22 thru 0xa19bcc5 (0x5fa4 bytes) 2024-04-24T00:36:46.546 INFO:teuthology.orchestra.run.smithi064.stdout:4980 clone 35 order 20 su 524288 sc 7 2024-04-24T00:36:46.632 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:46.632 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:46.633 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:46.627+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:46.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:46.734 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:46.731+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:46.819 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:46.819 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:46.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:46.835 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:46.831+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:46.936 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:46.931+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:46.957 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:46.957 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:46.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:47.037 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:47.031+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:47.138 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:47.135+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:47.144 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:47.144 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:47.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:47.239 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:47.235+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:47.335 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:47.335 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:47.340 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:47.339+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:47.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:47.441 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:47.435+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:47.477 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:47.477 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:47.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:47.542 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:47.539+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:47.642 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:47.639+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:47.653 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:47.653 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:47.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:47.744 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:47.738+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:47.844 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:47.844 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:47.844 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:47.838+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:47.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:47.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:47.953 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:47.951+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:47.978 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:47.978 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:47.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:48.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:36:48.059 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:48.059 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:48.060 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:48.055+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:48.084 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:48.084 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:48.121 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:48.121 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:48.149 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:48.150 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:48.151 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:48.143+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:48.181 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:48.220 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:48.221 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.00993180099999, 2024-04-24T00:36:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:36:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:36:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.00958088600001, 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:36:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:36:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:36:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.00269256600001, 2024-04-24T00:36:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:36:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.74441093999999, 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:36:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:36:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.734134524, 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.272183721000005, 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:36:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:36:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81718 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.259775454999996, 2024-04-24T00:36:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042549629999999998, 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81718 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961917+0000", 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961918+0000", 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4000000000000002e-07 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961920+0000", 2024-04-24T00:36:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7070000000000001e-06 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961928+0000", 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9039999999999992e-06 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961935+0000", 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.472e-06 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.003920+0000", 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041984974000000001 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004014+0000", 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4715000000000004e-05 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004467+0000", 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000452418 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81719 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 229376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.259743701000005, 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042736426000000001, 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81719 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961949+0000", 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9200000000000002e-07 2024-04-24T00:36:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961952+0000", 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7049999999999999e-06 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961967+0000", 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4846e-05 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.961977+0000", 2024-04-24T00:36:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0502999999999999e-05 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004498+0000", 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042520943999999998 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004583+0000", 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4449000000000004e-05 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004685+0000", 2024-04-24T00:36:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102587 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81734 3.8 3:15e8936a:::rbd_data.115e805f80b3.000000000000000c:2a [read 2965504~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.256211205, 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042293453000000002, 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81734 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965481+0000", 2024-04-24T00:36:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965482+0000", 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965484+0000", 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6189999999999998e-06 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965491+0000", 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2310000000000001e-06 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.965496+0000", 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1839999999999998e-06 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007617+0000", 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042121416000000002 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007688+0000", 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0310999999999998e-05 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.007775+0000", 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7269999999999996e-05 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.246149700000004, 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.032206793, 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:36:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:36:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:36:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.022194849000002, 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.748815987, 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:36:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:36:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:36:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.732147538000007, 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:36:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:36:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.546249754999998, 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:36:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.862491398000003, 2024-04-24T00:36:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:36:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:36:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:36:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:36:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.840274399999998, 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:36:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91840 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.679053119999999, 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046961500000000003, 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91840 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542639+0000", 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542641+0000", 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.184e-06 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542647+0000", 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6760000000000001e-06 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542664+0000", 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6379e-05 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542672+0000", 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0390000000000005e-06 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542752+0000", 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0475000000000005e-05 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.542858+0000", 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105722 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.547803+0000", 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004945245 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589123+0000", 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041320085999999999 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589275+0000", 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015216200000000001 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589454+0000", 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178214 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.589601+0000", 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014731799999999999 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.535704561999999, 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:36:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:36:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.523079602000003, 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:36:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:36:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:48.323 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:48.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:48.325 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:48.325 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:48.326 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:48.327 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:48.327 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:48.319+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:48.425 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:48.423+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:48.457 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:48.458 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:48.458 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:48.458 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:48.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:48.527 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:48.522+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:48.593 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:48.593 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:48.593 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:48.593 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:48.594 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:48.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:48.649 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:48.646+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:48.729 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:48.723+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:48.772 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:48.772 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:48.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:48.772 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:48.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.772 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:48.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:48.772 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.553522044, 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:48.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.13173635000001, 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:48.779 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:48.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:48.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:48.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:48.785 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.989608783999998, 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:48.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:48.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:48.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:48.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:48.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:48.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:48.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.851837001999996, 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.794 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:48.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:48.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.814354683999994, 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:48.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:48.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:48.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.587763819000003, 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.808 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.814 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.152008820000006, 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:48.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:48.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.142150783999995, 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:48.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:48.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:48.828 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 71.121801176000005, 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:48.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:48.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:48.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.087755841000003, 2024-04-24T00:36:48.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:48.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:48.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.085446005000001, 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.840 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:48.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:48.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.076877205999999, 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.847 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:48.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.853 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.137728113000001, 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.854 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:48.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:48.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:48.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:48.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:48.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:48.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:48.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:48.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:48.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:48.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:48.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:48.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:48.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:48.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:48.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:48.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:48.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:48.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:48.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:48.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.863 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.863 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.248338117000003, 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:48.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.204862319999997, 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:48.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:48.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.230889017999999, 2024-04-24T00:36:48.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:48.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:48.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.211747115000001, 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.885 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.252459561999999, 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:48.890 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:48.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.218457140000002, 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:48.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:126987 3.11 3:8f8290b4:::rbd_data.115ee44a9909.0000000000000003:46 [read 522240~2048 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.054003442, 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077109041000000003, 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:48.901 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 126987 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2500000000000005e-07 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713852+0000", 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713859+0000", 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0720000000000002e-06 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713864+0000", 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2900000000000002e-06 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790095+0000", 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076231499999999994 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:48.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790170+0000", 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4986000000000003e-05 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790953+0000", 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00078209500000000001 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:48.906 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:48.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:48.909 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:48.903+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:49.009 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:49.003+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:49.028 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:36:49.041 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:49.041 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:49.041 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:49.041 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:49.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:49.110 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:49.107+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:49.179 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:49.179 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:49.179 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:49.179 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:49.179 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:49.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:49.237 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:49.234+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:49.241 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-04-24T00:36:49.311 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:49.306+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:49.324 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:49.324 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.61264944499999, 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.325 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:36:49.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:36:49.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.61255152000001, 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:36:49.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:36:49.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 128.25317198799999, 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.336 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:36:49.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:36:49.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:36:49.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.84242786199999, 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:49.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.817159256, 2024-04-24T00:36:49.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.963955536, 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.351 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:36:49.356 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.931350421999994, 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:49.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:49.360 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.526775728000004, 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:36:49.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:36:49.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:36:49.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:36:49.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.468935497999993, 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.369 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.627373149, 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:49.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:36:49.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:36:49.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:36:49.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.648625598000002, 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:49.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:36:49.379 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.618983544999999, 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:36:49.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.472468134000003, 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.383 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:36:49.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:36:49.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.683695575999998, 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:36:49.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:36:49.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.396 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.660552162999998, 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:49.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:36:49.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.787243414999999, 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:36:49.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:36:49.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:36:49.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.406 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.734561138, 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:36:49.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127274 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.6534762790000004, 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071400004000000003, 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.411 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127274 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667542+0000", 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667545+0000", 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9059999999999998e-06 2024-04-24T00:36:49.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667553+0000", 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4910000000000005e-06 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667560+0000", 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5339999999999996e-06 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738552+0000", 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070991992000000004 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738615+0000", 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3559000000000006e-05 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738941+0000", 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032590199999999998 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.415 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127275 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.6534485649999997, 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071850050999999998, 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.416 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127275 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4000000000000002e-07 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667572+0000", 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9780000000000001e-06 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667581+0000", 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7080000000000003e-06 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667587+0000", 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738971+0000", 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071383263000000002 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739053+0000", 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2591999999999998e-05 2024-04-24T00:36:49.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739419+0000", 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036566600000000001 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:128046 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5920200580000001, 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072176436999999996, 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.420 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 128046 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728998+0000", 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8299999999999998e-07 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729000+0000", 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2479999999999999e-06 2024-04-24T00:36:49.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729006+0000", 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9569999999999999e-06 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729010+0000", 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4730000000000002e-06 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800122+0000", 2024-04-24T00:36:49.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071111555000000007 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800239+0000", 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117401 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.801174+0000", 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093442000000000004 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.424 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.425 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.425 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.425 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:49.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:49.427 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:49.422+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:49.482 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:49.482 INFO:teuthology.orchestra.run.smithi064.stdout:4983 write 0x89430e4 thru 0x894c1b2 (0x90cf bytes) 2024-04-24T00:36:49.507 INFO:teuthology.orchestra.run.smithi064.stdout:4984 write 0x79c14bc thru 0x79c83e7 (0x6f2c bytes) 2024-04-24T00:36:49.519 INFO:teuthology.orchestra.run.smithi064.stdout:4985 read 0x941b1b0 thru 0x941cf69 (0x1dba bytes) 2024-04-24T00:36:49.519 INFO:teuthology.orchestra.run.smithi064.stdout:4986 punch from 0x4bfceb1 to 0x4c0142e, (0x457d bytes) 2024-04-24T00:36:49.519 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:49.526 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:49.522+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:49.532 INFO:teuthology.orchestra.run.smithi064.stdout:4989 trunc from 0xd8d8e04 to 0x4b48b2e 2024-04-24T00:36:49.555 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:49.555 INFO:teuthology.orchestra.run.smithi064.stdout:4991 read 0x3a15a35 thru 0x3a1796d (0x1f39 bytes) 2024-04-24T00:36:49.565 INFO:teuthology.orchestra.run.smithi064.stdout:4992 trunc from 0x4b48b2e to 0xa146f53 2024-04-24T00:36:49.584 INFO:teuthology.orchestra.run.smithi064.stdout:4993 punch from 0x5065387 to 0x5072464, (0xd0dd bytes) 2024-04-24T00:36:49.584 INFO:teuthology.orchestra.run.smithi064.stdout:4995 read 0x8969927 thru 0x8971196 (0x7870 bytes) 2024-04-24T00:36:49.584 INFO:teuthology.orchestra.run.smithi064.stdout:4996 read 0x9ca41fe thru 0x9cad524 (0x9327 bytes) 2024-04-24T00:36:49.591 INFO:teuthology.orchestra.run.smithi064.stdout:4997 trunc from 0xa146f53 to 0x6ef3d6f 2024-04-24T00:36:49.612 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:49.613 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:49.613 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:49.613 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:49.627 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:49.628 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:49.623+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:49.635 INFO:teuthology.orchestra.run.smithi064.stdout:4998 writesame 0xbebc572 thru 0xbebc7e7 data_size 0xe(0x276 bytes) 2024-04-24T00:36:49.652 INFO:teuthology.orchestra.run.smithi064.stdout:4999 writesame 0xd9273a6 thru 0xd92a55a data_size 0x19(0x31b5 bytes) 2024-04-24T00:36:49.653 INFO:teuthology.orchestra.run.smithi064.stdout:5000 writesame 0x38ad417 thru 0x38b2c15 data_size 0x3(0x57ff bytes) 2024-04-24T00:36:49.689 INFO:teuthology.orchestra.run.smithi064.stdout:5004 writesame 0x3256f30 thru 0x325dcb0 data_size 0x11(0x6d81 bytes) 2024-04-24T00:36:49.697 INFO:teuthology.orchestra.run.smithi064.stdout:5005 writesame 0x5874b34 thru 0x5883f1b data_size 0x46(0xf3e8 bytes) 2024-04-24T00:36:49.707 INFO:teuthology.orchestra.run.smithi064.stdout:5006 write 0x7f51d42 thru 0x7f5e2dc (0xc59b bytes) 2024-04-24T00:36:49.720 INFO:teuthology.orchestra.run.smithi064.stdout:5007 read 0x6d5ab45 thru 0x6d69c2b (0xf0e7 bytes) 2024-04-24T00:36:49.720 INFO:teuthology.orchestra.run.smithi064.stdout:5011 write 0xc7d1001 thru 0xc7d2293 (0x1293 bytes) 2024-04-24T00:36:49.726 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:49.727 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:49.722+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:49.742 INFO:teuthology.orchestra.run.smithi064.stdout:5013 trunc from 0xd92a55b to 0x7d7dfc2 2024-04-24T00:36:49.790 INFO:teuthology.orchestra.run.smithi064.stdout:5015 write 0xc9fd78d thru 0xca08e6b (0xb6df bytes) 2024-04-24T00:36:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:5016 trunc from 0xca08e6c to 0xb18f365 2024-04-24T00:36:49.808 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:49.808 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:49.808 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:49.808 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:49.808 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:49.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:49.827 INFO:teuthology.orchestra.run.smithi064.stdout:5017 write 0x797d582 thru 0x797fa4c (0x24cb bytes) 2024-04-24T00:36:49.834 INFO:teuthology.orchestra.run.smithi064.stdout:5018 punch from 0x58827a3 to 0x5889d4b, (0x75a8 bytes) 2024-04-24T00:36:49.834 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:49.834 INFO:teuthology.orchestra.run.smithi064.stdout:5023 writesame 0x60548b1 thru 0x60626ed data_size 0x1(0xde3d bytes) 2024-04-24T00:36:49.865 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:49.862+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:49.929 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:49.927+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:49.993 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.45713011000001, 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:49.994 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:49.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:49.999 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.42309305699999, 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:36:50.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.003 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 119.307979061, 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:50.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:36:50.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:36:50.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 119.2805291, 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.009 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:36:50.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.360457268000005, 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.013 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:36:50.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:36:50.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.354186010000006, 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.018 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:36:50.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:36:50.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:50.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.342731423999993, 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:36:50.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.288278880000007, 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.028 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:36:50.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:36:50.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105171 3.1e 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head [zero 0~262144,write 262144~131072 [fadvise_sequential+fadvise_nocache],zero 393216~131072,write 524288~917504 [fadvise_sequential+fadvise_nocache],zero 1441792~393216,write 1835008~131072 [fadvise_sequential+fadvise_nocache],write 2097152~917504 [fadvise_sequential+fadvise_nocache],zero 3014656~393216,write 3407872~131072 [fadvise_sequential+fadvise_nocache],zero 3538944~131072,write 3670016~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.298528388000001, 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061400402, 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.033 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105171 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690858+0000", 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5059999999999998e-06 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697752+0000", 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006893766 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697795+0000", 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2589000000000002e-05 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697810+0000", 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5620999999999998e-05 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697936+0000", 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012529199999999999 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698150+0000", 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021430200000000001 2024-04-24T00:36:50.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698445+0000", 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029468099999999998 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736712+0000", 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038267781000000001 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:50.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752153+0000", 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015441043999999999 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752204+0000", 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752256+0000", 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2286e-05 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.038 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127047 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 374784~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650011+0000", 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.339373342, 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055335291000000002, 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.039 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127047 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650011+0000", 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650011+0000", 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650012+0000", 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7400000000000002e-07 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650016+0000", 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1980000000000003e-06 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650028+0000", 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.169e-05 2024-04-24T00:36:50.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650036+0000", 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7360000000000007e-06 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.704970+0000", 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054933370000000002 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705049+0000", 2024-04-24T00:36:50.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9080999999999997e-05 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705346+0000", 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029734199999999997 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127048 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 378880~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3393357420000003, 2024-04-24T00:36:50.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768986, 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127048 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:50.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.92e-07 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650053+0000", 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.698e-06 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650065+0000", 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1454000000000001e-05 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650073+0000", 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8920000000000006e-06 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705378+0000", 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055304820999999997 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705456+0000", 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8009999999999993e-05 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705818+0000", 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036131899999999997 2024-04-24T00:36:50.047 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127049 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 382976~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3392985700000004, 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056235535000000003, 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127049 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650090+0000", 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650101+0000", 2024-04-24T00:36:50.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0822000000000001e-05 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650110+0000", 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705851+0000", 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055741377000000002 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705916+0000", 2024-04-24T00:36:50.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5160000000000006e-05 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706321+0000", 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040486500000000002 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127050 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 387072~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.339263109, 2024-04-24T00:36:50.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056659432000000003, 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127050 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650122+0000", 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650125+0000", 2024-04-24T00:36:50.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.366e-06 2024-04-24T00:36:50.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650136+0000", 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0624e-05 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650144+0000", 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4360000000000002e-06 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706354+0000", 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056210006999999999 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706437+0000", 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2515000000000005e-05 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706781+0000", 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034373599999999998 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.059 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127051 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3392291570000001, 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057055097999999999, 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127051 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650156+0000", 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1399999999999996e-07 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650159+0000", 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3100000000000001e-06 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650170+0000", 2024-04-24T00:36:50.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.058e-05 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650178+0000", 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5660000000000007e-06 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706827+0000", 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056648506000000001 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706903+0000", 2024-04-24T00:36:50.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6493000000000001e-05 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.707210+0000", 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306929 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127334 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664445+0000", 2024-04-24T00:36:50.064 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3249395440000002, 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055399756000000001, 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127334 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664445+0000", 2024-04-24T00:36:50.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664445+0000", 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664445+0000", 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9599999999999999e-07 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664448+0000", 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3240000000000001e-06 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664454+0000", 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5919999999999997e-06 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664463+0000", 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1290000000000002e-06 2024-04-24T00:36:50.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719463+0000", 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054999271000000002 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719519+0000", 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6206999999999999e-05 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719845+0000", 2024-04-24T00:36:50.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032573699999999999 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127335 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3249135409999999, 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055806472000000003, 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127335 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:50.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5200000000000002e-07 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664474+0000", 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6400000000000001e-06 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664481+0000", 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.002e-06 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664493+0000", 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2311e-05 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719879+0000", 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055386187000000003 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719956+0000", 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6196e-05 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720277+0000", 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032168399999999998 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127336 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:50.073 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3248810290000002, 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056212985, 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127336 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664504+0000", 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5400000000000001e-07 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664513+0000", 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.132e-06 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664520+0000", 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3849999999999996e-06 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664525+0000", 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0180000000000001e-06 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720371+0000", 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055845976999999998 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720430+0000", 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8964e-05 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720716+0000", 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028595500000000001 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127337 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3248496559999996, 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056616423999999999, 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127337 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.078 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2099999999999997e-07 2024-04-24T00:36:50.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664538+0000", 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0410000000000002e-06 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664546+0000", 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3710000000000002e-06 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664551+0000", 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0889999999999999e-06 2024-04-24T00:36:50.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720759+0000", 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056208658000000002 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720811+0000", 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1564999999999998e-05 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721151+0000", 2024-04-24T00:36:50.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034017900000000002 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127338 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3248255159999998, 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057296623999999997, 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.082 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127338 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.58e-07 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664562+0000", 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.345e-06 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664568+0000", 2024-04-24T00:36:50.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664573+0000", 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721197+0000", 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056624102000000003 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721296+0000", 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9001999999999998e-05 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721855+0000", 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055934300000000001 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127339 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:50.086 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.3248049699999997, 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057818889999999998, 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127339 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:50.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664580+0000", 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5299999999999999e-07 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664582+0000", 2024-04-24T00:36:50.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1399999999999998e-06 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664588+0000", 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1959999999999996e-06 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664593+0000", 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.656e-06 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:50.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721894+0000", 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057301501999999997 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721991+0000", 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6330999999999996e-05 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.722398+0000", 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040761200000000001 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.090 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.091 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:50.091 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:50.091 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:50.096 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:50.096 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:50.090+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:50.096 INFO:teuthology.orchestra.run.smithi064.stdout:5025 writesame 0xe696684 thru 0xe69f8fa data_size 0x1(0x9277 bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5027 read 0x734ffea thru 0x73548d1 (0x48e8 bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5028 writesame 0x9abccc3 thru 0x9ac98a3 data_size 0x29(0xcbe1 bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5030 read 0x4bc3e1f thru 0x4bc595f (0x1b41 bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5031 writesame 0x7eb4e9d thru 0x7eb728e data_size 0x2(0x23f2 bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5032 trunc from 0xe69f8fb to 0xbf98071 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5033 punch from 0xa5e51d0 to 0xa5ed1c2, (0x7ff2 bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5034 read 0xbe6d4c thru 0xbf4269 (0xd51e bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5035 write 0x8a64ce3 thru 0x8a698e4 (0x4c02 bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5036 read 0x358f960 thru 0x359a30e (0xa9af bytes) 2024-04-24T00:36:50.097 INFO:teuthology.orchestra.run.smithi064.stdout:5037 write 0x310eb52 thru 0x311a25d (0xb70c bytes) 2024-04-24T00:36:50.100 INFO:teuthology.orchestra.run.smithi064.stdout:5038 read 0x40ff84e thru 0x4107f4c (0x86ff bytes) 2024-04-24T00:36:50.100 INFO:teuthology.orchestra.run.smithi064.stdout:5041 read 0x68ade96 thru 0x68b8a60 (0xabcb bytes) 2024-04-24T00:36:50.101 INFO:teuthology.orchestra.run.smithi064.stdout:5042 write 0x3dd80f9 thru 0x3dd9266 (0x116e bytes) 2024-04-24T00:36:50.108 INFO:teuthology.orchestra.run.smithi064.stdout:5043 punch from 0x59b0458 to 0x59b96f7, (0x929f bytes) 2024-04-24T00:36:50.108 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:50.121 INFO:teuthology.orchestra.run.smithi064.stdout:5047 write 0xc997706 thru 0xc99f530 (0x7e2b bytes) 2024-04-24T00:36:50.130 INFO:teuthology.orchestra.run.smithi064.stdout:5048 trunc from 0xc99f531 to 0xcfe7df7 2024-04-24T00:36:50.142 INFO:teuthology.orchestra.run.smithi064.stdout:5049 write 0xb7e77b9 thru 0xb7edff4 (0x683c bytes) 2024-04-24T00:36:50.151 INFO:teuthology.orchestra.run.smithi064.stdout:5051 write 0x9ffc1cc thru 0xa008558 (0xc38d bytes) 2024-04-24T00:36:50.173 INFO:teuthology.orchestra.run.smithi064.stdout:5053 trunc from 0xcfe7df7 to 0x823cb5b 2024-04-24T00:36:50.196 INFO:teuthology.orchestra.run.smithi064.stdout:5054 write 0x2155718 thru 0x2164c23 (0xf50c bytes) 2024-04-24T00:36:50.196 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:50.191+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:50.219 INFO:teuthology.orchestra.run.smithi064.stdout:5056 trunc from 0x823cb5b to 0xa0ead6 2024-04-24T00:36:50.269 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:50.269 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:50.269 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:50.269 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:50.274 INFO:teuthology.orchestra.run.smithi064.stdout:5057 punch from 0x80e862 to 0x80f6b2, (0xe50 bytes) 2024-04-24T00:36:50.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:50.307 INFO:teuthology.orchestra.run.smithi064.stdout:5059 writesame 0x7f5ecd4 thru 0x7f6b840 data_size 0x1(0xcb6d bytes) 2024-04-24T00:36:50.307 INFO:teuthology.orchestra.run.smithi064.stdout:5060 read 0x4446042 thru 0x444a3ee (0x43ad bytes) 2024-04-24T00:36:50.307 INFO:teuthology.orchestra.run.smithi064.stdout:5061 read 0x1ef3b35 thru 0x1efd4f3 (0x99bf bytes) 2024-04-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi064.stdout:5062 writesame 0x9336d4d thru 0x933752c data_size 0x6(0x7e0 bytes) 2024-04-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi064.stdout:5064 read 0x7422294 thru 0x7429e5f (0x7bcc bytes) 2024-04-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi064.stdout:5065 writesame 0x52a6ffa thru 0x52a77f9 data_size 0x10(0x800 bytes) 2024-04-24T00:36:50.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:50.329 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:50.326+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:50.341 INFO:teuthology.orchestra.run.smithi064.stdout:5066 trunc from 0x933752d to 0xd4fb9db 2024-04-24T00:36:50.354 INFO:teuthology.orchestra.run.smithi064.stdout:5067 read 0xac7b8fe thru 0xac7bc47 (0x34a bytes) 2024-04-24T00:36:50.354 INFO:teuthology.orchestra.run.smithi064.stdout:5068 write 0x2838c9f thru 0x283f883 (0x6be5 bytes) 2024-04-24T00:36:50.361 INFO:teuthology.orchestra.run.smithi064.stdout:5070 write 0x8521597 thru 0x8529342 (0x7dac bytes) 2024-04-24T00:36:50.369 INFO:teuthology.orchestra.run.smithi064.stdout:5071 writesame 0x4cf2be6 thru 0x4cfb29c data_size 0x1(0x86b7 bytes) 2024-04-24T00:36:50.400 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:50.394+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:50.441 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:50.441 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:50.441 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:50.442 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:50.442 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:50.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:50.477 INFO:teuthology.orchestra.run.smithi064.stdout:5072 punch from 0x1819980 to 0x181c8b0, (0x2f30 bytes) 2024-04-24T00:36:50.483 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:50.483 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:50.494 INFO:teuthology.orchestra.run.smithi064.stdout:5073 trunc from 0xd4fb9db to 0x1b5bb4c 2024-04-24T00:36:50.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:50.501 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:50.513 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:50.510+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:50.519 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:50.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:50.524 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5, 0] out_osds: [7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:50.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:50.525 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [1, 2, 3, 4, 6, 5, 0] 2024-04-24T00:36:50.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-04-24T00:36:50.545 INFO:teuthology.orchestra.run.smithi064.stdout:5074 writesame 0xa7e1c3e thru 0xa7e6934 data_size 0x13(0x4cf7 bytes) 2024-04-24T00:36:50.545 INFO:teuthology.orchestra.run.smithi064.stdout:5075 write 0x4b74306 thru 0x4b7432b (0x26 bytes) 2024-04-24T00:36:50.546 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:50.546 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:50.552 INFO:teuthology.orchestra.run.smithi064.stdout:5076 read 0x1821e4c thru 0x18280ec (0x62a1 bytes) 2024-04-24T00:36:50.564 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:50.564 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:50.569 INFO:teuthology.orchestra.run.smithi064.stdout:5077 write 0xd931b19 thru 0xd937d49 (0x6231 bytes) 2024-04-24T00:36:50.574 INFO:teuthology.orchestra.run.smithi064.stdout:5078 write 0x779629d thru 0x77a4a4c (0xe7b0 bytes) 2024-04-24T00:36:50.582 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:50.582 INFO:teuthology.orchestra.run.smithi064.stdout:5080 punch from 0x2df5927 to 0x2df827f, (0x2958 bytes) 2024-04-24T00:36:50.582 INFO:teuthology.orchestra.run.smithi064.stdout:5081 punch from 0xbbe06c6 to 0xbbe22c6, (0x1c00 bytes) 2024-04-24T00:36:50.582 INFO:teuthology.orchestra.run.smithi064.stdout:5082 punch from 0x7b2a769 to 0x7b3715e, (0xc9f5 bytes) 2024-04-24T00:36:50.583 INFO:teuthology.orchestra.run.smithi064.stdout:5084 trunc from 0xd937d4a to 0xc6b7ca5 2024-04-24T00:36:50.584 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:50.585 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:50.602 INFO:teuthology.orchestra.run.smithi064.stdout:5085 writesame 0xb39870c thru 0xb3a5454 data_size 0x1(0xcd49 bytes) 2024-04-24T00:36:50.602 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:50.598+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:50.604 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:50.605 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:50.622 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:50.623 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:50.641 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:50.641 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:50.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:50.705 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:50.702+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:50.741 INFO:teuthology.orchestra.run.smithi064.stdout:5086 write 0x5c07249 thru 0x5c14948 (0xd700 bytes) 2024-04-24T00:36:50.753 INFO:teuthology.orchestra.run.smithi064.stdout:5087 read 0x33434c3 thru 0x3349c69 (0x67a7 bytes) 2024-04-24T00:36:50.753 INFO:teuthology.orchestra.run.smithi064.stdout:5088 writesame 0x306a7a9 thru 0x3070ca5 data_size 0x1(0x64fd bytes) 2024-04-24T00:36:50.804 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:50.804 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:50.804 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:50.799+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:50.806 INFO:teuthology.orchestra.run.smithi064.stdout:5089 read 0xc5fc47 thru 0xc601f9 (0x5b3 bytes) 2024-04-24T00:36:50.807 INFO:teuthology.orchestra.run.smithi064.stdout:5090 read 0x82a9668 thru 0x82b8c72 (0xf60b bytes) 2024-04-24T00:36:50.809 INFO:teuthology.orchestra.run.smithi064.stdout:5091 trunc from 0xc6b7ca5 to 0xb4a2fb2 2024-04-24T00:36:50.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:50.822 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:50.822 INFO:teuthology.orchestra.run.smithi064.stdout:5093 read 0x62c2a21 thru 0x62c7f2f (0x550f bytes) 2024-04-24T00:36:50.826 INFO:teuthology.orchestra.run.smithi064.stdout:5094 trunc from 0xb4a2fb2 to 0xebf909e 2024-04-24T00:36:50.839 INFO:teuthology.orchestra.run.smithi064.stdout:5096 read 0x26c1773 thru 0x26c634b (0x4bd9 bytes) 2024-04-24T00:36:50.839 INFO:teuthology.orchestra.run.smithi064.stdout:5097 read 0x930ddf1 thru 0x931cbcf (0xeddf bytes) 2024-04-24T00:36:50.839 INFO:teuthology.orchestra.run.smithi064.stdout:5098 write 0x8c55b2 thru 0x8d1357 (0xbda6 bytes) 2024-04-24T00:36:50.847 INFO:teuthology.orchestra.run.smithi064.stdout:5099 read 0xb12b120 thru 0xb13571b (0xa5fc bytes) 2024-04-24T00:36:50.847 INFO:teuthology.orchestra.run.smithi064.stdout:5100 write 0xe646940 thru 0xe652770 (0xbe31 bytes) 2024-04-24T00:36:50.853 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:36:50.847+0000 7f841a5f8640 -1 mon.a@0(leader).osd e183 definitely_dead 0 2024-04-24T00:36:50.860 INFO:teuthology.orchestra.run.smithi064.stdout:5101 writesame 0xd5a3b18 thru 0xd5b0e21 data_size 0x22(0xd30a bytes) 2024-04-24T00:36:50.869 INFO:teuthology.orchestra.run.smithi064.stdout:5102 read 0x19ee933 thru 0x19f933b (0xaa09 bytes) 2024-04-24T00:36:50.884 INFO:teuthology.orchestra.run.smithi064.stdout:5103 trunc from 0xebf909e to 0x46095d5 2024-04-24T00:36:50.904 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:50.899+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:50.931 INFO:teuthology.orchestra.run.smithi064.stdout:5104 writesame 0x66788a1 thru 0x667f31c data_size 0x1d(0x6a7c bytes) 2024-04-24T00:36:50.954 INFO:teuthology.orchestra.run.smithi064.stdout:5106 write 0xa663a78 thru 0xa6679d0 (0x3f59 bytes) 2024-04-24T00:36:50.955 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:50.955 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:50.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:50.982 INFO:teuthology.orchestra.run.smithi064.stdout:5107 trunc from 0xa6679d1 to 0xd07034 2024-04-24T00:36:51.005 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:51.002+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:51.049 INFO:teuthology.orchestra.run.smithi064.stdout:5109 writesame 0xcb12ca4 thru 0xcb12fdf data_size 0xc(0x33c bytes) 2024-04-24T00:36:51.049 INFO:teuthology.orchestra.run.smithi064.stdout:5111 punch from 0x384cce6 to 0x38582ce, (0xb5e8 bytes) 2024-04-24T00:36:51.049 INFO:teuthology.orchestra.run.smithi064.stdout:5112 read 0x44578aa thru 0x4464cb2 (0xd409 bytes) 2024-04-24T00:36:51.049 INFO:teuthology.orchestra.run.smithi064.stdout:5113 read 0x94b3e19 thru 0x94b7c9f (0x3e87 bytes) 2024-04-24T00:36:51.049 INFO:teuthology.orchestra.run.smithi064.stdout:5114 read 0x2276235 thru 0x2280685 (0xa451 bytes) 2024-04-24T00:36:51.050 INFO:teuthology.orchestra.run.smithi064.stdout:5117 punch from 0x28358ea to 0x284163d, (0xbd53 bytes) 2024-04-24T00:36:51.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:36:51.065 INFO:teuthology.orchestra.run.smithi064.stdout:5118 writesame 0xd4e6e9a thru 0xd4e8476 data_size 0x1d(0x15dd bytes) 2024-04-24T00:36:51.065 INFO:teuthology.orchestra.run.smithi064.stdout:5119 read 0xb2f4380 thru 0xb2fc94d (0x85ce bytes) 2024-04-24T00:36:51.065 INFO:teuthology.orchestra.run.smithi064.stdout:5120 punch from 0xcce53e7 to 0xccf343c, (0xe055 bytes) 2024-04-24T00:36:51.065 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:51.065 INFO:teuthology.orchestra.run.smithi064.stdout:5122 read 0xa282ab9 thru 0xa2864e7 (0x3a2f bytes) 2024-04-24T00:36:51.065 INFO:teuthology.orchestra.run.smithi064.stdout:5123 writesame 0x6468bc thru 0x655d27 data_size 0x2(0xf46c bytes) 2024-04-24T00:36:51.106 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:51.103+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:51.122 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:51.122 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:51.127 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:51.128 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:51.128 INFO:teuthology.orchestra.run.smithi064.stdout:5129 read 0x1e9ea50 thru 0x1ea89ac (0x9f5d bytes) 2024-04-24T00:36:51.128 INFO:teuthology.orchestra.run.smithi064.stdout:5130 write 0xd0086ea thru 0xd012b81 (0xa498 bytes) 2024-04-24T00:36:51.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:51.141 INFO:teuthology.orchestra.run.smithi064.stdout:5131 read 0x4bd9912 thru 0x4bdd4d4 (0x3bc3 bytes) 2024-04-24T00:36:51.141 INFO:teuthology.orchestra.run.smithi064.stdout:5134 write 0x1729314 thru 0x172c570 (0x325d bytes) 2024-04-24T00:36:51.149 INFO:teuthology.orchestra.run.smithi064.stdout:5135 read 0x5eaa48d thru 0x5eb94bf (0xf033 bytes) 2024-04-24T00:36:51.149 INFO:teuthology.orchestra.run.smithi064.stdout:5136 read 0x825e64c thru 0x8262165 (0x3b1a bytes) 2024-04-24T00:36:51.149 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:51.149 INFO:teuthology.orchestra.run.smithi064.stdout:5138 writesame 0x82ed3d2 thru 0x82f9c6c data_size 0x1(0xc89b bytes) 2024-04-24T00:36:51.207 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:51.202+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:51.266 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:51.266 INFO:teuthology.orchestra.run.smithi064.stdout:5140 writesame 0x18b901e thru 0x18bd8cd data_size 0x8(0x48b0 bytes) 2024-04-24T00:36:51.275 INFO:teuthology.orchestra.run.smithi064.stdout:5141 writesame 0x3885560 thru 0x388b311 data_size 0x43(0x5db2 bytes) 2024-04-24T00:36:51.301 INFO:teuthology.orchestra.run.smithi064.stdout:5142 trunc from 0xd4e8477 to 0x4dddc56 2024-04-24T00:36:51.308 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:51.302+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:51.311 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:51.311 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:51.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:51.351 INFO:teuthology.orchestra.run.smithi064.stdout:5143 write 0x53b42c9 thru 0x53c21fe (0xdf36 bytes) 2024-04-24T00:36:51.379 INFO:teuthology.orchestra.run.smithi064.stdout:5145 read 0x3ffd2cc thru 0x4002491 (0x51c6 bytes) 2024-04-24T00:36:51.392 INFO:teuthology.orchestra.run.smithi064.stdout:5146 write 0x5bbcd21 thru 0x5bc39fd (0x6cdd bytes) 2024-04-24T00:36:51.405 INFO:teuthology.orchestra.run.smithi064.stdout:5147 writesame 0x282e641 thru 0x28348ff data_size 0x1(0x62bf bytes) 2024-04-24T00:36:51.409 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:51.403+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:51.443 INFO:teuthology.orchestra.run.smithi064.stdout:5148 read 0x1d371de thru 0x1d3f286 (0x80a9 bytes) 2024-04-24T00:36:51.443 INFO:teuthology.orchestra.run.smithi064.stdout:5149 writesame 0x2f67332 thru 0x2f6fe85 data_size 0x4a(0x8b54 bytes) 2024-04-24T00:36:51.473 INFO:teuthology.orchestra.run.smithi064.stdout:5150 writesame 0x5cb3e51 thru 0x5cbb304 data_size 0x61(0x74b4 bytes) 2024-04-24T00:36:51.479 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:51.479 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:51.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:51.495 INFO:teuthology.orchestra.run.smithi064.stdout:5151 write 0x7898410 thru 0x78a2dec (0xa9dd bytes) 2024-04-24T00:36:51.502 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:51.502 INFO:teuthology.orchestra.run.smithi064.stdout:5154 read 0x7304938 thru 0x730f438 (0xab01 bytes) 2024-04-24T00:36:51.502 INFO:teuthology.orchestra.run.smithi064.stdout:5155 clone 36 order 19 su 2048 sc 8 2024-04-24T00:36:51.509 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:51.507+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:51.579 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.0. 2024-04-24T00:36:51.579 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:36:51.611 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:51.606+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:51.627 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:51.628 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:51.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:51.711 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:51.706+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:51.804 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:51.804 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:51.813 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:51.806+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:51.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:51.914 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:51.910+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:51.948 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:51.948 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:51.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:52.014 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:52.011+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:52.089 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:52.089 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:52.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:52.115 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:52.110+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:52.216 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:52.211+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:52.274 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:52.274 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:52.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:52.317 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:52.314+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:52.418 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:52.415+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:52.458 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:52.458 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:52.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:52.519 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:52.514+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:52.620 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:52.614+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:52.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:52.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:52.721 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:52.718+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:52.784 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:52.784 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:52.800 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:52.800 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:52.807 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:52.807 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:52.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:52.817 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:52.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:52.822 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:52.818+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:52.834 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:52.834 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:52.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:52.853 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:52.871 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:52.871 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:52.890 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:52.890 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:52.907 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:52.908 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:52.922 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:52.919+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:52.991 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:52.991 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:52.991 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:52.991 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:52.991 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:53.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:53.022 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:53.019+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:53.123 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:53.118+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:53.133 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:53.133 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:53.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:53.133 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:53.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.133 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:53.133 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 159.91472844399999, 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:53.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:53.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.49294275, 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:53.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:53.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.350815183999998, 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:53.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.213043401999997, 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:53.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:53.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:53.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.157 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 79.175561083999995, 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:53.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.163 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 78.948970219000003, 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.513215220000006, 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.169 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:53.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:53.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.503357183999995, 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:53.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 75.483007576000006, 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.181 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:53.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:53.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.448962241000004, 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:53.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:53.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.446652404999995, 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.191 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:53.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:53.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:53.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:53.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:53.196 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.438083606000006, 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:53.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.498934513000002, 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:53.202 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:53.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:53.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:53.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.609544517000003, 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:53.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.566068719999997, 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.592095418, 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.572953515000002, 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.224 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:53.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:53.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:53.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.613665961999999, 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.228 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:53.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:53.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.579663539999999, 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.234 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:53.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:53.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:126987 3.11 3:8f8290b4:::rbd_data.115ee44a9909.0000000000000003:46 [read 522240~2048 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.4152098419999994, 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077109041000000003, 2024-04-24T00:36:53.238 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 126987 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:53.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2500000000000005e-07 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713852+0000", 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713859+0000", 2024-04-24T00:36:53.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0720000000000002e-06 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713864+0000", 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2900000000000002e-06 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790095+0000", 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076231499999999994 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790170+0000", 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4986000000000003e-05 2024-04-24T00:36:53.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790953+0000", 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00078209500000000001 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.242 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:53.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:53.244 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:53.239+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:53.344 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:53.338+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:53.412 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:53.412 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:53.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:53.445 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:53.439+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:53.545 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:53.542+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:53.594 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:53.594 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:53.594 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:53.594 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:53.595 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:53.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:36:53.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:53.657 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:53.655+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:53.747 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:53.743+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:53.812 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:53.812 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:53.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:53.812 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:53.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.812 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:53.812 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:53.812 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.10101257700001, 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:53.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:36:53.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:36:53.817 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.100914652, 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.818 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:53.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:36:53.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:36:53.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.74153512000001, 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.825 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:36:53.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:36:53.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:53.830 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.330790994, 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:36:53.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:36:53.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:53.836 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.305522388, 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:53.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.452318668000004, 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:36:53.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:36:53.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:53.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:53.846 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.419713553999998, 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.015138859999993, 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:36:53.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:53.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:53.856 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.957298629999997, 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.860 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.115736280999997, 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:36:53.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:36:53.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.136988729999999, 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.865 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:53.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:36:53.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:36:53.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:36:53.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 48.107346677000002, 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:36:53.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:36:53.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.960831266, 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.875 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:36:53.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:53.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:53.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:53.881 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.172058708000002, 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:36:53.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:36:53.887 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.148915295000002, 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:36:53.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.891 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.275606547000002, 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:36:53.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:36:53.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:53.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:36:53.897 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.22292427, 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.901 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127274 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.1418394109999994, 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071400004000000003, 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127274 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667542+0000", 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:36:53.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667545+0000", 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9059999999999998e-06 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667553+0000", 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4910000000000005e-06 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667560+0000", 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5339999999999996e-06 2024-04-24T00:36:53.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738552+0000", 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070991992000000004 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738615+0000", 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3559000000000006e-05 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738941+0000", 2024-04-24T00:36:53.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032590199999999998 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127275 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.1418116969999996, 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071850050999999998, 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.906 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127275 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:53.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4000000000000002e-07 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667572+0000", 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9780000000000001e-06 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667581+0000", 2024-04-24T00:36:53.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7080000000000003e-06 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667587+0000", 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738971+0000", 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071383263000000002 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739053+0000", 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2591999999999998e-05 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739419+0000", 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036566600000000001 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:128046 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:53.910 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.0803831899999992, 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072176436999999996, 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 128046 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:53.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728998+0000", 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8299999999999998e-07 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:53.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729000+0000", 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2479999999999999e-06 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729006+0000", 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9569999999999999e-06 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729010+0000", 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4730000000000002e-06 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:53.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800122+0000", 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071111555000000007 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800239+0000", 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117401 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.801174+0000", 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093442000000000004 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.914 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.915 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:53.915 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:53.915 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:53.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:53.917 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:53.910+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:54.017 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:54.014+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:54.087 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:54.087 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:54.087 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:54.087 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:54.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:54.118 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:54.115+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:54.219 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:54.214+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:54.277 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:54.277 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:54.277 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:54.277 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:54.277 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:54.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:54.337 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:54.334+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:54.421 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:54.415+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:54.479 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:54.479 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:54.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:54.479 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:54.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.479 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.94290896300001, 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:36:54.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:36:54.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:36:54.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.90887190999999, 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:36:54.485 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:36:54.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.793757914, 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.489 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:36:54.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.76630795299999, 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.498 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.846236121000004, 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:54.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:36:54.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:36:54.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.839964863000006, 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:36:54.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:36:54.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.828510277000007, 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:36:54.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 76.774057733000006, 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105171 3.1e 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head [zero 0~262144,write 262144~131072 [fadvise_sequential+fadvise_nocache],zero 393216~131072,write 524288~917504 [fadvise_sequential+fadvise_nocache],zero 1441792~393216,write 1835008~131072 [fadvise_sequential+fadvise_nocache],write 2097152~917504 [fadvise_sequential+fadvise_nocache],zero 3014656~393216,write 3407872~131072 [fadvise_sequential+fadvise_nocache],zero 3538944~131072,write 3670016~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.784307241, 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061400402, 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.517 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105171 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690858+0000", 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5059999999999998e-06 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697752+0000", 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006893766 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697795+0000", 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2589000000000002e-05 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697810+0000", 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5620999999999998e-05 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697936+0000", 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012529199999999999 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698150+0000", 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021430200000000001 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698445+0000", 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029468099999999998 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736712+0000", 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038267781000000001 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752153+0000", 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015441043999999999 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752204+0000", 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752256+0000", 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2286e-05 2024-04-24T00:36:54.522 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127048 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 378880~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.8251145950000005, 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768986, 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.523 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127048 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.92e-07 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650053+0000", 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.698e-06 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650065+0000", 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1454000000000001e-05 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650073+0000", 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8920000000000006e-06 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705378+0000", 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055304820999999997 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705456+0000", 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8009999999999993e-05 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705818+0000", 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036131899999999997 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127049 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 382976~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.8250774229999998, 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056235535000000003, 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.527 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127049 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650090+0000", 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650101+0000", 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0822000000000001e-05 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650110+0000", 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705851+0000", 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055741377000000002 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705916+0000", 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5160000000000006e-05 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706321+0000", 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040486500000000002 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127050 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 387072~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.531 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.8250419620000002, 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056659432000000003, 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127050 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650122+0000", 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650125+0000", 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.366e-06 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650136+0000", 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0624e-05 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650144+0000", 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4360000000000002e-06 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706354+0000", 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056210006999999999 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706437+0000", 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2515000000000005e-05 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706781+0000", 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034373599999999998 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.535 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127051 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.8250080099999995, 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057055097999999999, 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127051 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650156+0000", 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1399999999999996e-07 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650159+0000", 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3100000000000001e-06 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650170+0000", 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.058e-05 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.538 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650178+0000", 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5660000000000007e-06 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706827+0000", 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056648506000000001 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706903+0000", 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6493000000000001e-05 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.707210+0000", 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306929 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127335 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.8106923940000001, 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055806472000000003, 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.540 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127335 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5200000000000002e-07 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664474+0000", 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6400000000000001e-06 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664481+0000", 2024-04-24T00:36:54.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.002e-06 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664493+0000", 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2311e-05 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719879+0000", 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055386187000000003 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719956+0000", 2024-04-24T00:36:54.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6196e-05 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720277+0000", 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032168399999999998 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127336 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.8106598819999995, 2024-04-24T00:36:54.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056212985, 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127336 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664504+0000", 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5400000000000001e-07 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664513+0000", 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.132e-06 2024-04-24T00:36:54.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664520+0000", 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3849999999999996e-06 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664525+0000", 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0180000000000001e-06 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720371+0000", 2024-04-24T00:36:54.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055845976999999998 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720430+0000", 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8964e-05 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720716+0000", 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028595500000000001 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127337 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.8106285090000007, 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056616423999999999, 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127337 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2099999999999997e-07 2024-04-24T00:36:54.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664538+0000", 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0410000000000002e-06 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664546+0000", 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3710000000000002e-06 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664551+0000", 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0889999999999999e-06 2024-04-24T00:36:54.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720759+0000", 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056208658000000002 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720811+0000", 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1564999999999998e-05 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721151+0000", 2024-04-24T00:36:54.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034017900000000002 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127338 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.810604369, 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057296623999999997, 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.553 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127338 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.58e-07 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664562+0000", 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.345e-06 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664568+0000", 2024-04-24T00:36:54.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664573+0000", 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721197+0000", 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056624102000000003 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721296+0000", 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9001999999999998e-05 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721855+0000", 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055934300000000001 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127339 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:54.557 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.810583823, 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057818889999999998, 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127339 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:54.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664580+0000", 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5299999999999999e-07 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664582+0000", 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1399999999999998e-06 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664588+0000", 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1959999999999996e-06 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664593+0000", 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.656e-06 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721894+0000", 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057301501999999997 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721991+0000", 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6330999999999996e-05 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.722398+0000", 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040761200000000001 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.561 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.0992090980000002, 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:36:54.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:36:54.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:36:54.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:54.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:36:54.567 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.0783175519999997, 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:36:54.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:54.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:54.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:54.571 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:54.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:54.573 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:54.570+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:54.574 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:36:54.673 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:54.670+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:54.746 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:54.747 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:54.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:54.809 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:54.806+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:54.875 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:54.870+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:54.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:54.940 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:54.940 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:54.940 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:54.940 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:54.940 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:54.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:54.998 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:54.994+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:55.077 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:55.074+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:55.096 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:55.097 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:55.114 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:55.128 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:55.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 194.19953951100001, 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:36:55.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:36:55.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 161.26757382299999, 2024-04-24T00:36:55.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:36:55.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:36:55.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:36:55.141 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 161.266325075, 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:55.142 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:36:55.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:36:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:36:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:36:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:36:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:36:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:36:55.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:36:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:36:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:36:55.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:36:55.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 152.10870097099999, 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:55.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:36:55.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:36:55.156 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 152.09520642800001, 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:36:55.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:36:55.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 147.83342531299999, 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.161 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:36:55.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:36:55.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:55.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:36:55.167 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 138.57983917000001, 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:36:55.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:36:55.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 138.55216530600001, 2024-04-24T00:36:55.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:36:55.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:36:55.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:36:55.178 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 134.05814248600001, 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:36:55.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:55.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.184 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 134.04913365199999, 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:36:55.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.547434704, 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.189 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:36:55.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.193 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.547372358, 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.194 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:36:55.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:36:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:36:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:36:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.546862679, 2024-04-24T00:36:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:36:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.199 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:36:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:36:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:36:55.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:36:55.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:36:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:36:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:36:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:36:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:36:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:36:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:36:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:36:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:36:55.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:36:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:36:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:36:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:36:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:36:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.546751617, 2024-04-24T00:36:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:36:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.208 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:36:55.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.213 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.54668635100001, 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:36:55.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:36:55.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.546626285, 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:36:55.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:36:55.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.546569592, 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.223 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:36:55.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:36:55.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.546500259, 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:36:55.227 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:36:55.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.54640963999999, 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:36:55.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.54636622699999, 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:55.237 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:36:55.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:36:55.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.242 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:55.242 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:55.242 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:55.243 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:55.244 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:55.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:55.245 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:55.238+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:55.245 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:55.246 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:55.246 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:55.246 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:55.246 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:55.247 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:55.247 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:55.247 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:55.344 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:55.338+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:55.377 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:55.378 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:55.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:36:55.445 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:55.443+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:55.538 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:55.539 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:55.547 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:55.543+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:55.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:36:55.647 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:55.643+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:55.749 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:55.743+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:55.764 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:55.764 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:55.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:36:55.870 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:55.867+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:55.943 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:55.943 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:55.950 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:55.946+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:55.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:36:56.051 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:56.047+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:56.120 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:56.120 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:56.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:36:56.182 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:56.179+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:56.254 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:56.251+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:56.310 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:56.311 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:56.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:36:56.370 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:56.367+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:56.456 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:56.450+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:56.496 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:56.496 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:56.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:36:56.556 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:56.550+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:56.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:36:56.611 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:36:56.612 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:36:56.612 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-24T00:36:56.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T00:36:56.649 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:56.649 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:56.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:36:56.666 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:56.663+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:56.758 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:56.754+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:56.859 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:56.854+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:56.874 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:56.874 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:56.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:36:56.960 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:56.955+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:57.002 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:36:57.002 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 189, 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "6a70d87d-4453-47be-af2a-5590f9851254", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-04-24T00:33:24.857711+0000", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-04-24T00:36:56.574299+0000", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-04-24T00:35:14.574643+0000", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-04-24T00:36:51.571099+0000", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5806080, 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-04-24T00:36:57.003 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 5, 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 3, 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 8, 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-04-24T00:36:57.004 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:39.734814+0000", 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:57.005 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:57.006 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "17", 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:57.007 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:57.008 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:57.009 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 8, 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 8, 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.25, 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-04-24T00:36:57.010 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:43.786311+0000", 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:57.011 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:57.012 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "20", 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-04-24T00:36:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:57.015 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 1.5, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 1.5, 2024-04-24T00:36:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "pool_client.0", 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-04-24T00:33:51.288702+0000", 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 2, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 1, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-04-24T00:36:57.017 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 32, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 32, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 32, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 32, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 32, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-04-24T00:36:57.018 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "187", 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 75, 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 187, 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-04-24T00:36:57.019 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-04-24T00:36:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-04-24T00:36:57.021 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.75, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.75, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.75, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.059999942779541, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.059999942779541, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.91740989685058594, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.92000001668930054, 2024-04-24T00:36:57.022 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "1c80b3f4-99ee-4725-90a8-920116161bf5", 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 13, 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 74, 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 95, 2024-04-24T00:36:57.023 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 187, 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 75, 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6817", 2024-04-24T00:36:57.024 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6818", 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6819", 2024-04-24T00:36:57.025 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6822", 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.026 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6823", 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6820", 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.027 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6821", 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 218259947 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6817/218259947", 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6819/218259947", 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6823/218259947", 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6821/218259947", 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "20e1331a-f8a2-48fd-b2b8-b7a92f21d78e", 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.264190673828125, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 159, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6824", 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6825", 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6826", 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6827", 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6830", 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6831", 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.032 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6828", 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6829", 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 637720411 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6825/637720411", 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6827/637720411", 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6831/637720411", 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6829/637720411", 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:57.033 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "a51557a0-9634-4a20-82b0-b54cdd36b166", 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 188, 2024-04-24T00:36:57.034 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.037 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 820024667 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.038 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/820024667", 2024-04-24T00:36:57.039 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6811/820024667", 2024-04-24T00:36:57.039 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6815/820024667", 2024-04-24T00:36:57.039 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6813/820024667", 2024-04-24T00:36:57.039 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:57.039 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:57.039 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:57.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.041 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "aafffa79-f532-4441-b1d6-f6441c5eebf8", 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.79296875, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.3392791748046875, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 184, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:57.042 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-04-24T00:36:57.045 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1974534323 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/1974534323", 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6803/1974534323", 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1974534323", 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1974534323", 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:57.046 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "78f05556-598e-4be2-b682-e3f58863eb82", 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.9623870849609375, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:57.047 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 188, 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6816", 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6818", 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6820", 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6821", 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6826", 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6828", 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6823", 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6824", 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1642543604 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6818/1642543604", 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6821/1642543604", 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6828/1642543604", 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6824/1642543604", 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5", 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:57.052 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 188, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6808", 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:57.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6809", 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6810", 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:57.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6811", 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6814", 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6815", 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6812", 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.056 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6813", 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 172726194 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6809/172726194", 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6811/172726194", 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6815/172726194", 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6813/172726194", 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.057 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "6dad12bf-94de-482a-a402-49e67d5e5dc7", 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 187, 2024-04-24T00:36:57.058 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6800", 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6801", 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:57.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6802", 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6803", 2024-04-24T00:36:57.060 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6806", 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6807", 2024-04-24T00:36:57.061 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6804", 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6805", 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2195746987 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6801/2195746987", 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6803/2195746987", 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6807/2195746987", 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6805/2195746987", 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:36:57.063 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "e6734cfe-3c24-4ddd-8ef0-5ed450efe772", 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 13, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 62, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6817", 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6819", 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6822", 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6825", 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6830", 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6831", 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-24T00:36:57.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6827", 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.167:6829", 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 954791674 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.167:6819/954791674", 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.167:6825/954791674", 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.167:6831/954791674", 2024-04-24T00:36:57.068 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.167:6829/954791674", 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:57.069 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.509233+0000", 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 75 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.465696+0000", 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:57.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.405177+0000", 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:57.071 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.269202+0000", 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.359180+0000", 2024-04-24T00:36:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.351834+0000", 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-04-24T00:36:57.073 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.490127+0000", 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-04-24T00:36:57.074 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540701547738038271, 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-04-24T00:33:33.484842+0000", 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.16", 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: 2 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.1b", 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-04-24T00:36:57.076 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [ 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "snaps": [ 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 71, 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:36:57.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "begin": 73, 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "length": 1 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-04-24T00:36:57.078 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-04-24T00:36:57.079 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-04-24T00:36:57.079 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-04-24T00:36:57.079 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-04-24T00:36:57.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:36:57.079 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:36:57.081 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:57.074+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:57.082 INFO:tasks.thrashosds.thrasher:j is {'epoch': 189, 'fsid': '6a70d87d-4453-47be-af2a-5590f9851254', 'created': '2024-04-24T00:33:24.857711+0000', 'modified': '2024-04-24T00:36:56.574299+0000', 'last_up_change': '2024-04-24T00:35:14.574643+0000', 'last_in_change': '2024-04-24T00:36:51.571099+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T00:33:39.734814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T00:33:43.786311+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-24T00:33:51.288702+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 75, 'snap_epoch': 187, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 0.75, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9174098968505859, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '1c80b3f4-99ee-4725-90a8-920116161bf5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 74, 'up_from': 95, 'up_thru': 187, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 218259947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 218259947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 218259947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 218259947}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 218259947}]}, 'public_addr': '172.21.15.64:6817/218259947', 'cluster_addr': '172.21.15.64:6819/218259947', 'heartbeat_back_addr': '172.21.15.64:6823/218259947', 'heartbeat_front_addr': '172.21.15.64:6821/218259947', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '20e1331a-f8a2-48fd-b2b8-b7a92f21d78e', 'up': 1, 'in': 1, 'weight': 0.264190673828125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 159, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 637720411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 637720411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 637720411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 637720411}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 637720411}]}, 'public_addr': '172.21.15.64:6825/637720411', 'cluster_addr': '172.21.15.64:6827/637720411', 'heartbeat_back_addr': '172.21.15.64:6831/637720411', 'heartbeat_front_addr': '172.21.15.64:6829/637720411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a51557a0-9634-4a20-82b0-b54cdd36b166', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 188, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 820024667}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 820024667}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 820024667}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 820024667}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 820024667}]}, 'public_addr': '172.21.15.64:6809/820024667', 'cluster_addr': '172.21.15.64:6811/820024667', 'heartbeat_back_addr': '172.21.15.64:6815/820024667', 'heartbeat_front_addr': '172.21.15.64:6813/820024667', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aafffa79-f532-4441-b1d6-f6441c5eebf8', 'up': 1, 'in': 1, 'weight': 0.79296875, 'primary_affinity': 0.3392791748046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 184, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1974534323}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1974534323}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1974534323}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1974534323}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1974534323}]}, 'public_addr': '172.21.15.64:6801/1974534323', 'cluster_addr': '172.21.15.64:6803/1974534323', 'heartbeat_back_addr': '172.21.15.64:6807/1974534323', 'heartbeat_front_addr': '172.21.15.64:6805/1974534323', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '78f05556-598e-4be2-b682-e3f58863eb82', 'up': 1, 'in': 1, 'weight': 0.9623870849609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 188, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6816', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6818', 'nonce': 1642543604}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6820', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6821', 'nonce': 1642543604}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6826', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6828', 'nonce': 1642543604}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6823', 'nonce': 1642543604}, {'type': 'v1', 'addr': '172.21.15.167:6824', 'nonce': 1642543604}]}, 'public_addr': '172.21.15.167:6818/1642543604', 'cluster_addr': '172.21.15.167:6821/1642543604', 'heartbeat_back_addr': '172.21.15.167:6828/1642543604', 'heartbeat_front_addr': '172.21.15.167:6824/1642543604', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 188, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6808', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 172726194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6810', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 172726194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6814', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 172726194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6812', 'nonce': 172726194}, {'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 172726194}]}, 'public_addr': '172.21.15.167:6809/172726194', 'cluster_addr': '172.21.15.167:6811/172726194', 'heartbeat_back_addr': '172.21.15.167:6815/172726194', 'heartbeat_front_addr': '172.21.15.167:6813/172726194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6dad12bf-94de-482a-a402-49e67d5e5dc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 187, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6800', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 2195746987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6802', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 2195746987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6806', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 2195746987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6804', 'nonce': 2195746987}, {'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 2195746987}]}, 'public_addr': '172.21.15.167:6801/2195746987', 'cluster_addr': '172.21.15.167:6803/2195746987', 'heartbeat_back_addr': '172.21.15.167:6807/2195746987', 'heartbeat_front_addr': '172.21.15.167:6805/2195746987', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e6734cfe-3c24-4ddd-8ef0-5ed450efe772', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6817', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6819', 'nonce': 954791674}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6822', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6825', 'nonce': 954791674}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6830', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6831', 'nonce': 954791674}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6827', 'nonce': 954791674}, {'type': 'v1', 'addr': '172.21.15.167:6829', 'nonce': 954791674}]}, 'public_addr': '172.21.15.167:6819/954791674', 'cluster_addr': '172.21.15.167:6825/954791674', 'heartbeat_back_addr': '172.21.15.167:6831/954791674', 'heartbeat_front_addr': '172.21.15.167:6829/954791674', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.509233+0000', 'dead_epoch': 75}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.465696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.405177+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.269202+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.359180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.351834+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.490127+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T00:33:33.484842+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.16', 'osds': [3, 2]}, {'pgid': '3.1b', 'osds': [5, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 71, 'length': 1}, {'begin': 73, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T00:36:57.082 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-24T00:36:57.083 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:57.083 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:57.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:36:57.181 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:36:57.178+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:36:57.248 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:57.248 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:57.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:36:57.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:36:57.306 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:57.303+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:57.383 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:57.379+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:57.390 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:57.391 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:57.407 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:57.408 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:57.424 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:57.424 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:57.427 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:57.427 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:57.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:36:57.441 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:57.441 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:57.460 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:57.460 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:57.477 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:57.484 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:57.479+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:57.498 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:57.516 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:57.516 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:57.573 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:57.573 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:57.573 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:57.573 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:57.583 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:36:57.586 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:57.582+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:57.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:36:57.686 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:36:57.682+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:36:57.787 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:57.783+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:57.794 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:57.794 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:57.794 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:57.794 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:57.795 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:57.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:36:57.888 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:57.883+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:57.989 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:57.987+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:58.001 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:58.001 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:58.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:58.001 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:58.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.001 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:36:58.001 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 164.78489558199999, 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.002 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:36:58.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:36:58.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.006 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:36:58.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.363109888, 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:36:58.008 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:58.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:36:58.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:36:58.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:36:58.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:36:58.014 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 89.220982321999998, 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:36:58.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:36:58.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:36:58.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.083210539999996, 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.022 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:36:58.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:36:58.028 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.045728221999994, 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.029 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:36:58.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 83.819137357000002, 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:36:58.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:36:58.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:36:58.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:36:58.042 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.383382358000006, 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.043 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:36:58.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:36:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.373524321999994, 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:36:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:36:58.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 80.353174714000005, 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:36:58.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:36:58.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:36:58.063 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.319129379000003, 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:36:58.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.316819542999994, 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:36:58.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:36:58.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:36:58.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.308250744000006, 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.076 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:36:58.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.082 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.369101650999994, 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:36:58.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:36:58.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.479711655000003, 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:36:58.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:36:58.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.436235858000003, 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:36:58.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.462262555999999, 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.102 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:36:58.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:36:58.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.443120653000001, 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.108 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:36:58.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:36:58.112 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.483833099999998, 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:36:58.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.118 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.449830677999998, 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:36:58.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:36:58.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:126987 3.11 3:8f8290b4:::rbd_data.115ee44a9909.0000000000000003:46 [read 522240~2048 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.285376980000001, 2024-04-24T00:36:58.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077109041000000003, 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 126987 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2500000000000005e-07 2024-04-24T00:36:58.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713852+0000", 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713859+0000", 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0720000000000002e-06 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713864+0000", 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2900000000000002e-06 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790095+0000", 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076231499999999994 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790170+0000", 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4986000000000003e-05 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790953+0000", 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00078209500000000001 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.128 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:58.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:36:58.130 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:58.127+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:58.230 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:58.226+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:58.262 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:58.262 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:58.262 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:58.262 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:58.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:36:58.361 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:58.358+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:58.432 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:58.427+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:58.439 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:58.439 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:58.439 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:58.439 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:58.439 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:58.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:36:58.545 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:58.542+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:58.628 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.915984372, 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.91588644699999, 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:36:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:36:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:36:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 137.556506915, 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:36:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.145762789, 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.12049418300001, 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.26729046299999, 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:36:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.234685349, 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:36:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:36:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:36:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 99.830110654999999, 2024-04-24T00:36:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:36:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 99.772270425000002, 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:36:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.930708076000002, 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:36:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:36:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.951960524999997, 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:36:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:36:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.922318472000001, 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:36:58.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.775803060999998, 2024-04-24T00:36:58.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:36:58.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:36:58.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:36:58.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:36:58.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.987030503, 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:36:58.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:36:58.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:36:58.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.96388709, 2024-04-24T00:36:58.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:36:58.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:36:58.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.090578342000001, 2024-04-24T00:36:58.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 38.037896064999998, 2024-04-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127274 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.956811205999999, 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071400004000000003, 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127274 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:58.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667542+0000", 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667545+0000", 2024-04-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9059999999999998e-06 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667553+0000", 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4910000000000005e-06 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667560+0000", 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5339999999999996e-06 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738552+0000", 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070991992000000004 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738615+0000", 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3559000000000006e-05 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738941+0000", 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032590199999999998 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127275 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.956783492, 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071850050999999998, 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127275 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4000000000000002e-07 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667572+0000", 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9780000000000001e-06 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667581+0000", 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7080000000000003e-06 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667587+0000", 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738971+0000", 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071383263000000002 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739053+0000", 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2591999999999998e-05 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739419+0000", 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036566600000000001 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:128046 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.895354985000001, 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072176436999999996, 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:58.728 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 128046 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728998+0000", 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8299999999999998e-07 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729000+0000", 2024-04-24T00:36:58.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2479999999999999e-06 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729006+0000", 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9569999999999999e-06 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729010+0000", 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4730000000000002e-06 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800122+0000", 2024-04-24T00:36:58.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071111555000000007 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800239+0000", 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117401 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.801174+0000", 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093442000000000004 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:58.732 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:58.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:36:58.735 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:58.731+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:58.807 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #34, image image_client.0-clone34 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent35 2024-04-24T00:36:58.834 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:58.831+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:58.903 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:58.904 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:58.904 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:58.904 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:58.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:36:58.969 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:58.958+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:59.037 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:59.031+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:59.052 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:59.052 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:59.052 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:59.052 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:59.052 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:59.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:36:59.137 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:59.135+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:59.238 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:36:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.702125752, 2024-04-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:36:59.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:36:59.245 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.66808869900001, 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:36:59.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:36:59.249 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 128.55297470299999, 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.250 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:36:59.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:36:59.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:36:59.256 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 128.52552474199999, 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:36:59.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:36:59.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:36:59.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.260 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.605452909999997, 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.261 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:36:59.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.599181651999999, 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.266 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.271 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.587727065999999, 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:36:59.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:36:59.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 81.533274521999999, 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.276 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:36:59.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.280 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105171 3.1e 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head [zero 0~262144,write 262144~131072 [fadvise_sequential+fadvise_nocache],zero 393216~131072,write 524288~917504 [fadvise_sequential+fadvise_nocache],zero 1441792~393216,write 1835008~131072 [fadvise_sequential+fadvise_nocache],write 2097152~917504 [fadvise_sequential+fadvise_nocache],zero 3014656~393216,write 3407872~131072 [fadvise_sequential+fadvise_nocache],zero 3538944~131072,write 3670016~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.54352403, 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061400402, 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105171 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690858+0000", 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5059999999999998e-06 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697752+0000", 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006893766 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697795+0000", 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2589000000000002e-05 2024-04-24T00:36:59.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697810+0000", 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5620999999999998e-05 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697936+0000", 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012529199999999999 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698150+0000", 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021430200000000001 2024-04-24T00:36:59.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698445+0000", 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029468099999999998 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736712+0000", 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038267781000000001 2024-04-24T00:36:59.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752153+0000", 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015441043999999999 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752204+0000", 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752256+0000", 2024-04-24T00:36:59.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2286e-05 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127048 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 378880~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.584331384, 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768986, 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127048 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:59.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.92e-07 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650053+0000", 2024-04-24T00:36:59.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.698e-06 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650065+0000", 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1454000000000001e-05 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650073+0000", 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8920000000000006e-06 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705378+0000", 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055304820999999997 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705456+0000", 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8009999999999993e-05 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705818+0000", 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036131899999999997 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127049 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 382976~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.584294212, 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056235535000000003, 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127049 2024-04-24T00:36:59.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650090+0000", 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650101+0000", 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0822000000000001e-05 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650110+0000", 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705851+0000", 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055741377000000002 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705916+0000", 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5160000000000006e-05 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706321+0000", 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040486500000000002 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127050 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 387072~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.584258751, 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056659432000000003, 2024-04-24T00:36:59.298 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127050 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650122+0000", 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650125+0000", 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.366e-06 2024-04-24T00:36:59.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650136+0000", 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0624e-05 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650144+0000", 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4360000000000002e-06 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706354+0000", 2024-04-24T00:36:59.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056210006999999999 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706437+0000", 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2515000000000005e-05 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706781+0000", 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034373599999999998 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127051 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.584224798999999, 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057055097999999999, 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127051 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650156+0000", 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1399999999999996e-07 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650159+0000", 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3100000000000001e-06 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650170+0000", 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.058e-05 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650178+0000", 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5660000000000007e-06 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706827+0000", 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056648506000000001 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706903+0000", 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6493000000000001e-05 2024-04-24T00:36:59.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.707210+0000", 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306929 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127335 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.569909183, 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055806472000000003, 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.307 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127335 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5200000000000002e-07 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664474+0000", 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6400000000000001e-06 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664481+0000", 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.002e-06 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664493+0000", 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2311e-05 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719879+0000", 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055386187000000003 2024-04-24T00:36:59.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719956+0000", 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6196e-05 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720277+0000", 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032168399999999998 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127336 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.569876670999999, 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056212985, 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127336 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664504+0000", 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5400000000000001e-07 2024-04-24T00:36:59.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664513+0000", 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.132e-06 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664520+0000", 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3849999999999996e-06 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664525+0000", 2024-04-24T00:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0180000000000001e-06 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720371+0000", 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055845976999999998 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720430+0000", 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8964e-05 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720716+0000", 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028595500000000001 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127337 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.569845298000001, 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056616423999999999, 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.316 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127337 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:59.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2099999999999997e-07 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664538+0000", 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0410000000000002e-06 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664546+0000", 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3710000000000002e-06 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664551+0000", 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0889999999999999e-06 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720759+0000", 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056208658000000002 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720811+0000", 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1564999999999998e-05 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721151+0000", 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034017900000000002 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127338 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:59.320 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.569821158, 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057296623999999997, 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127338 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.58e-07 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664562+0000", 2024-04-24T00:36:59.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.345e-06 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664568+0000", 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664573+0000", 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721197+0000", 2024-04-24T00:36:59.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056624102000000003 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721296+0000", 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9001999999999998e-05 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721855+0000", 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055934300000000001 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127339 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.569800612, 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057818889999999998, 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127339 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664580+0000", 2024-04-24T00:36:59.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5299999999999999e-07 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664582+0000", 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1399999999999998e-06 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664588+0000", 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1959999999999996e-06 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664593+0000", 2024-04-24T00:36:59.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.656e-06 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721894+0000", 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057301501999999997 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721991+0000", 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6330999999999996e-05 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.722398+0000", 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040761200000000001 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.8584258869999992, 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:36:59.329 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:36:59.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:36:59.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.8375343409999996, 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:36:59.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:36:59.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:36:59.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:36:59.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:36:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:36:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:36:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:36:59.339 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:36:59.339 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:59.339 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:59.234+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:59.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:36:59.341 INFO:teuthology.orchestra.run.smithi064.stdout:5156 read 0x2911921 thru 0x291947d (0x7b5d bytes) 2024-04-24T00:36:59.341 INFO:teuthology.orchestra.run.smithi064.stdout:5157 trunc from 0x78a2ded to 0x2e3ff85 2024-04-24T00:36:59.372 INFO:teuthology.orchestra.run.smithi064.stdout:5158 read 0x5f2eb0 thru 0x5fd2a0 (0xa3f1 bytes) 2024-04-24T00:36:59.374 INFO:teuthology.orchestra.run.smithi064.stdout:5159 read 0x16fa995 thru 0x17051e9 (0xa855 bytes) 2024-04-24T00:36:59.376 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.376 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.376 INFO:teuthology.orchestra.run.smithi064.stdout:5165 read 0x25b5abd thru 0x25bed5c (0x92a0 bytes) 2024-04-24T00:36:59.385 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:59.382+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:59.415 INFO:teuthology.orchestra.run.smithi064.stdout:5166 writesame 0xd237e37 thru 0xd23f0af data_size 0x1(0x7279 bytes) 2024-04-24T00:36:59.415 INFO:teuthology.orchestra.run.smithi064.stdout:5169 read 0xc90d2fd thru 0xc911a77 (0x477b bytes) 2024-04-24T00:36:59.415 INFO:teuthology.orchestra.run.smithi064.stdout:5170 read 0x215171a thru 0x215d256 (0xbb3d bytes) 2024-04-24T00:36:59.415 INFO:teuthology.orchestra.run.smithi064.stdout:5171 read 0xd9fce2 thru 0xda9a49 (0x9d68 bytes) 2024-04-24T00:36:59.416 INFO:teuthology.orchestra.run.smithi064.stdout:5172 punch from 0xb72fee0 to 0xb7328d1, (0x29f1 bytes) 2024-04-24T00:36:59.417 INFO:teuthology.orchestra.run.smithi064.stdout:5173 punch from 0x93daaf2 to 0x93dc1b7, (0x16c5 bytes) 2024-04-24T00:36:59.417 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.417 INFO:teuthology.orchestra.run.smithi064.stdout:5176 writesame 0x43bad7c thru 0x43bbcfc data_size 0x1b(0xf81 bytes) 2024-04-24T00:36:59.426 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.426 INFO:teuthology.orchestra.run.smithi064.stdout:5179 read 0x5065469 thru 0x50667f7 (0x138f bytes) 2024-04-24T00:36:59.426 INFO:teuthology.orchestra.run.smithi064.stdout:5181 read 0xac19868 thru 0xac1e73b (0x4ed4 bytes) 2024-04-24T00:36:59.426 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.426 INFO:teuthology.orchestra.run.smithi064.stdout:5183 write 0x33f909d thru 0x3403ac1 (0xaa25 bytes) 2024-04-24T00:36:59.443 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:36:59.438+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:36:59.479 INFO:teuthology.orchestra.run.smithi064.stdout:5184 writesame 0xe6a8b88 thru 0xe6b0f57 data_size 0x26(0x83d0 bytes) 2024-04-24T00:36:59.479 INFO:teuthology.orchestra.run.smithi064.stdout:5186 write 0x5ceae5c thru 0x5cf0eb1 (0x6056 bytes) 2024-04-24T00:36:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:5187 punch from 0xcc3c0c8 to 0xcc43ddc, (0x7d14 bytes) 2024-04-24T00:36:59.500 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.506 INFO:teuthology.orchestra.run.smithi064.stdout:5190 trunc from 0xe6b0f58 to 0xc574c8a 2024-04-24T00:36:59.519 INFO:teuthology.orchestra.run.smithi064.stdout:5191 write 0x738b81c thru 0x738df8d (0x2772 bytes) 2024-04-24T00:36:59.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:36:59.534 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:59.534 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:59.534 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:36:59.534 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:59.536 INFO:teuthology.orchestra.run.smithi064.stdout:5192 read 0xc0bad19 thru 0xc0c47a1 (0x9a89 bytes) 2024-04-24T00:36:59.536 INFO:teuthology.orchestra.run.smithi064.stdout:5193 writesame 0x311e705 thru 0x312601d data_size 0x1(0x7919 bytes) 2024-04-24T00:36:59.547 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.547 INFO:teuthology.orchestra.run.smithi064.stdout:5195 punch from 0x451ed53 to 0x4527e22, (0x90cf bytes) 2024-04-24T00:36:59.547 INFO:teuthology.orchestra.run.smithi064.stdout:5196 write 0x955f872 thru 0x956e43f (0xebce bytes) 2024-04-24T00:36:59.548 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:36:59.543+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:36:59.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:36:59.562 INFO:teuthology.orchestra.run.smithi064.stdout:5197 write 0x29da4ae thru 0x29e923e (0xed91 bytes) 2024-04-24T00:36:59.568 INFO:teuthology.orchestra.run.smithi064.stdout:5198 write 0x79ceffb thru 0x79d9684 (0xa68a bytes) 2024-04-24T00:36:59.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:36:59.615 INFO:teuthology.orchestra.run.smithi064.stdout:5200 trunc from 0xc574c8a to 0x11ddc36 2024-04-24T00:36:59.644 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:36:59.639+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:36:59.683 INFO:teuthology.orchestra.run.smithi064.stdout:5205 trunc from 0x11ddc36 to 0x8012984 2024-04-24T00:36:59.692 INFO:teuthology.orchestra.run.smithi064.stdout:5206 read 0x779c5b8 thru 0x779ef10 (0x2959 bytes) 2024-04-24T00:36:59.692 INFO:teuthology.orchestra.run.smithi064.stdout:5207 punch from 0x337e7ee to 0x33843c9, (0x5bdb bytes) 2024-04-24T00:36:59.692 INFO:teuthology.orchestra.run.smithi064.stdout:5208 read 0x7a4b9c7 thru 0x7a597f3 (0xde2d bytes) 2024-04-24T00:36:59.692 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.692 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.711 INFO:teuthology.orchestra.run.smithi064.stdout:5212 write 0xedad4f6 thru 0xedb25d6 (0x50e1 bytes) 2024-04-24T00:36:59.729 INFO:teuthology.orchestra.run.smithi064.stdout:5213 write 0xba07d60 thru 0xba08366 (0x607 bytes) 2024-04-24T00:36:59.734 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:36:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:36:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:36:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:36:59.735 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:36:59.737 INFO:teuthology.orchestra.run.smithi064.stdout:5215 read 0x5a9ea26 thru 0x5aa2b36 (0x4111 bytes) 2024-04-24T00:36:59.737 INFO:teuthology.orchestra.run.smithi064.stdout:5216 write 0x29abdec thru 0x29b4b91 (0x8da6 bytes) 2024-04-24T00:36:59.738 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:36:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:36:59.745 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:36:59.743+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:36:59.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi064.stdout:5217 read 0x9bdb9a5 thru 0x9bdeea7 (0x3503 bytes) 2024-04-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi064.stdout:5221 write 0x1bc530e thru 0x1bcc2aa (0x6f9d bytes) 2024-04-24T00:36:59.757 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:36:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:36:59.774 INFO:teuthology.orchestra.run.smithi064.stdout:5222 read 0xe9ad62e thru 0xe9bbca1 (0xe674 bytes) 2024-04-24T00:36:59.774 INFO:teuthology.orchestra.run.smithi064.stdout:5223 punch from 0xc3e0550 to 0xc3ec914, (0xc3c4 bytes) 2024-04-24T00:36:59.774 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.774 INFO:teuthology.orchestra.run.smithi064.stdout:5226 punch from 0x8ae579 to 0x8ba15b, (0xbbe2 bytes) 2024-04-24T00:36:59.774 INFO:teuthology.orchestra.run.smithi064.stdout:5228 read 0x786b8a7 thru 0x786dfde (0x2738 bytes) 2024-04-24T00:36:59.774 INFO:teuthology.orchestra.run.smithi064.stdout:5229 read 0x7331c0b thru 0x7341b63 (0xff59 bytes) 2024-04-24T00:36:59.774 INFO:teuthology.orchestra.run.smithi064.stdout:5231 read 0xb01d6db thru 0xb0276c2 (0x9fe8 bytes) 2024-04-24T00:36:59.774 INFO:teuthology.orchestra.run.smithi064.stdout:5232 write 0x87b7a51 thru 0x87baad3 (0x3083 bytes) 2024-04-24T00:36:59.775 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:36:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:36:59.788 INFO:teuthology.orchestra.run.smithi064.stdout:5235 write 0x13392e0 thru 0x13405b8 (0x72d9 bytes) 2024-04-24T00:36:59.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:36:59.794 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:36:59.801 INFO:teuthology.orchestra.run.smithi064.stdout:5236 read 0xb07793e thru 0xb07bf3a (0x45fd bytes) 2024-04-24T00:36:59.802 INFO:teuthology.orchestra.run.smithi064.stdout:5237 writesame 0x7a4bc04 thru 0x7a590a6 data_size 0x5f(0xd4a3 bytes) 2024-04-24T00:36:59.813 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:36:59.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:36:59.816 INFO:teuthology.orchestra.run.smithi064.stdout:5241 write 0x9dbd702 thru 0x9dc27f0 (0x50ef bytes) 2024-04-24T00:36:59.830 INFO:teuthology.orchestra.run.smithi064.stdout:5242 read 0x79f4c89 thru 0x7a01d50 (0xd0c8 bytes) 2024-04-24T00:36:59.831 INFO:teuthology.orchestra.run.smithi064.stdout:5245 write 0xc65d9c4 thru 0xc669f6a (0xc5a7 bytes) 2024-04-24T00:36:59.835 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:36:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:36:59.846 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:36:59.843+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:36:59.849 INFO:teuthology.orchestra.run.smithi064.stdout:5246 punch from 0x13fa384 to 0x13fba40, (0x16bc bytes) 2024-04-24T00:36:59.850 INFO:teuthology.orchestra.run.smithi064.stdout:5247 writesame 0x8c0d989 thru 0x8c1a466 data_size 0x17(0xcade bytes) 2024-04-24T00:36:59.854 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:36:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:36:59.866 INFO:teuthology.orchestra.run.smithi064.stdout:5248 write 0x6361d93 thru 0x636979b (0x7a09 bytes) 2024-04-24T00:36:59.873 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:36:59.873 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:36:59.890 INFO:teuthology.orchestra.run.smithi064.stdout:5249 punch from 0x2e24c69 to 0x2e2d51a, (0x88b1 bytes) 2024-04-24T00:36:59.890 INFO:teuthology.orchestra.run.smithi064.stdout:5250 read 0x9b76770 thru 0x9b7f4d5 (0x8d66 bytes) 2024-04-24T00:36:59.890 INFO:teuthology.orchestra.run.smithi064.stdout:5251 write 0x1bde529 thru 0x1bea1a3 (0xbc7b bytes) 2024-04-24T00:36:59.899 INFO:teuthology.orchestra.run.smithi064.stdout:5253 trunc from 0xedb25d7 to 0xde502da 2024-04-24T00:36:59.906 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:36:59.906 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:36:59.918 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:36:59.918 INFO:teuthology.orchestra.run.smithi064.stdout:5256 punch from 0x21cd42b to 0x21d7380, (0x9f55 bytes) 2024-04-24T00:36:59.918 INFO:teuthology.orchestra.run.smithi064.stdout:5257 write 0xcff98dd thru 0xcfff517 (0x5c3b bytes) 2024-04-24T00:36:59.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:36:59.933 INFO:teuthology.orchestra.run.smithi064.stdout:5260 read 0x395c0a8 thru 0x3966c46 (0xab9f bytes) 2024-04-24T00:36:59.940 INFO:teuthology.orchestra.run.smithi064.stdout:5261 trunc from 0xde502da to 0x9a211ab 2024-04-24T00:36:59.946 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:36:59.942+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:36:59.970 INFO:teuthology.orchestra.run.smithi064.stdout:5264 trunc from 0x9a211ab to 0x4f1c6d8 2024-04-24T00:36:59.991 INFO:teuthology.orchestra.run.smithi064.stdout:5265 write 0x3b2fadd thru 0x3b33aaf (0x3fd3 bytes) 2024-04-24T00:37:00.021 INFO:teuthology.orchestra.run.smithi064.stdout:5266 write 0xcfaa4e4 thru 0xcfb54b5 (0xafd2 bytes) 2024-04-24T00:37:00.041 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:00.042 INFO:teuthology.orchestra.run.smithi064.stdout:5271 punch from 0xb1a920b to 0xb1aea48, (0x583d bytes) 2024-04-24T00:37:00.048 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:00.043+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:5272 trunc from 0xcfb54b6 to 0x503646b 2024-04-24T00:37:00.060 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:00.060 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:00.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:00.078 INFO:teuthology.orchestra.run.smithi064.stdout:5274 punch from 0x3f63ed2 to 0x3f722f2, (0xe420 bytes) 2024-04-24T00:37:00.078 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:00.096 INFO:teuthology.orchestra.run.smithi064.stdout:5276 write 0xa401b1b thru 0xa40da65 (0xbf4b bytes) 2024-04-24T00:37:00.124 INFO:teuthology.orchestra.run.smithi064.stdout:5277 writesame 0x389c1c thru 0x397b81 data_size 0x5f(0xdf66 bytes) 2024-04-24T00:37:00.136 INFO:teuthology.orchestra.run.smithi064.stdout:5278 read 0x34cc674 thru 0x34da436 (0xddc3 bytes) 2024-04-24T00:37:00.136 INFO:teuthology.orchestra.run.smithi064.stdout:5279 read 0x1a28517 thru 0x1a2da02 (0x54ec bytes) 2024-04-24T00:37:00.138 INFO:teuthology.orchestra.run.smithi064.stdout:5281 write 0x6d89994 thru 0x6d92f89 (0x95f6 bytes) 2024-04-24T00:37:00.149 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:00.146+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:00.165 INFO:teuthology.orchestra.run.smithi064.stdout:5282 punch from 0x569cc4 to 0x56aff5, (0x1331 bytes) 2024-04-24T00:37:00.174 INFO:teuthology.orchestra.run.smithi064.stdout:5284 trunc from 0xa40da66 to 0x523c592 2024-04-24T00:37:00.201 INFO:teuthology.orchestra.run.smithi064.stdout:5285 read 0x218b3f1 thru 0x2194025 (0x8c35 bytes) 2024-04-24T00:37:00.209 INFO:teuthology.orchestra.run.smithi064.stdout:5286 trunc from 0x523c592 to 0x9ac3614 2024-04-24T00:37:00.210 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:00.210 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:00.220 INFO:teuthology.orchestra.run.smithi064.stdout:5287 punch from 0x64b8fd1 to 0x64be9d0, (0x59ff bytes) 2024-04-24T00:37:00.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:00.236 INFO:teuthology.orchestra.run.smithi064.stdout:5288 trunc from 0x9ac3614 to 0x93edf8 2024-04-24T00:37:00.251 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:00.246+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:00.287 INFO:teuthology.orchestra.run.smithi064.stdout:5291 read 0x8e5469 thru 0x8f3a00 (0xe598 bytes) 2024-04-24T00:37:00.315 INFO:teuthology.orchestra.run.smithi064.stdout:5292 write 0xa4202b4 thru 0xa42cb54 (0xc8a1 bytes) 2024-04-24T00:37:00.345 INFO:teuthology.orchestra.run.smithi064.stdout:5293 read 0x5ee7b13 thru 0x5eebd11 (0x41ff bytes) 2024-04-24T00:37:00.349 INFO:teuthology.orchestra.run.smithi064.stdout:5294 trunc from 0xa42cb55 to 0xcfe2c66 2024-04-24T00:37:00.350 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:00.347+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:00.366 INFO:teuthology.orchestra.run.smithi064.stdout:5296 trunc from 0xcfe2c66 to 0xa5d0c9b 2024-04-24T00:37:00.372 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:00.372 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:00.385 INFO:teuthology.orchestra.run.smithi064.stdout:5298 write 0x6624a3f thru 0x662793c (0x2efe bytes) 2024-04-24T00:37:00.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:00.418 INFO:teuthology.orchestra.run.smithi064.stdout:5299 read 0x548ef4d thru 0x5490e55 (0x1f09 bytes) 2024-04-24T00:37:00.431 INFO:teuthology.orchestra.run.smithi064.stdout:5302 write 0xc952c00 thru 0xc955a92 (0x2e93 bytes) 2024-04-24T00:37:00.451 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:00.447+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:00.458 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:00.458 INFO:teuthology.orchestra.run.smithi064.stdout:5305 write 0x861619d thru 0x861dcc8 (0x7b2c bytes) 2024-04-24T00:37:00.486 INFO:teuthology.orchestra.run.smithi064.stdout:5307 read 0x42c4316 thru 0x42c9166 (0x4e51 bytes) 2024-04-24T00:37:00.486 INFO:teuthology.orchestra.run.smithi064.stdout:5310 write 0x9023369 thru 0x9028ee1 (0x5b79 bytes) 2024-04-24T00:37:00.508 INFO:teuthology.orchestra.run.smithi064.stdout:5311 writesame 0xbf9fed2 thru 0xbfacb67 data_size 0x57(0xcc96 bytes) 2024-04-24T00:37:00.542 INFO:teuthology.orchestra.run.smithi064.stdout:5312 writesame 0xd83736b thru 0xd83eab0 data_size 0x15(0x7746 bytes) 2024-04-24T00:37:00.543 INFO:teuthology.orchestra.run.smithi064.stdout:5313 write 0x1b08d1c thru 0x1b09c81 (0xf66 bytes) 2024-04-24T00:37:00.545 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:00.545 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:00.552 INFO:teuthology.orchestra.run.smithi064.stdout:5314 writesame 0x46e15e3 thru 0x46e1603 data_size 0x1(0x21 bytes) 2024-04-24T00:37:00.552 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:00.546+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:00.558 INFO:teuthology.orchestra.run.smithi064.stdout:5315 write 0x12598e5 thru 0x1268792 (0xeeae bytes) 2024-04-24T00:37:00.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:00.576 INFO:teuthology.orchestra.run.smithi064.stdout:5316 read 0x7b7bfe9 thru 0x7b7f393 (0x33ab bytes) 2024-04-24T00:37:00.601 INFO:teuthology.orchestra.run.smithi064.stdout:5317 trunc from 0xd83eab1 to 0x2cbcc49 2024-04-24T00:37:00.637 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:37:00.652 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:00.646+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:00.687 INFO:teuthology.orchestra.run.smithi064.stdout:5318 writesame 0xe449e91 thru 0xe450c96 data_size 0x1(0x6e06 bytes) 2024-04-24T00:37:00.701 INFO:teuthology.orchestra.run.smithi064.stdout:5319 trunc from 0xe450c97 to 0x55652f5 2024-04-24T00:37:00.720 INFO:teuthology.orchestra.run.smithi064.stdout:5322 trunc from 0x55652f5 to 0xa0e3a72 2024-04-24T00:37:00.731 INFO:teuthology.orchestra.run.smithi064.stdout:5324 trunc from 0xa0e3a72 to 0xafe02bd 2024-04-24T00:37:00.732 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:00.733 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:00.743 INFO:teuthology.orchestra.run.smithi064.stdout:5325 punch from 0xe732a2 to 0xe7b51f, (0x827d bytes) 2024-04-24T00:37:00.743 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:00.743 INFO:teuthology.orchestra.run.smithi064.stdout:5327 write 0x8588c9 thru 0x860d3d (0x8475 bytes) 2024-04-24T00:37:00.749 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:00.753 INFO:teuthology.orchestra.run.smithi064.stdout:5329 write 0x7a88cac thru 0x7a960c0 (0xd415 bytes) 2024-04-24T00:37:00.754 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:00.751+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:00.796 INFO:teuthology.orchestra.run.smithi064.stdout:5331 write 0xe588dfb thru 0xe596f27 (0xe12d bytes) 2024-04-24T00:37:00.828 INFO:teuthology.orchestra.run.smithi064.stdout:5333 write 0xcf2cbe9 thru 0xcf3570b (0x8b23 bytes) 2024-04-24T00:37:00.854 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:00.851+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:00.855 INFO:teuthology.orchestra.run.smithi064.stdout:5334 write 0xbd9c3a7 thru 0xbd9f62e (0x3288 bytes) 2024-04-24T00:37:00.875 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:00.876 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:00.876 INFO:teuthology.orchestra.run.smithi064.stdout:5335 writesame 0x3fc52d7 thru 0x3fcadcc data_size 0x3(0x5af6 bytes) 2024-04-24T00:37:00.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:00.896 INFO:teuthology.orchestra.run.smithi064.stdout:5336 write 0xb0de6e4 thru 0xb0e35e8 (0x4f05 bytes) 2024-04-24T00:37:00.917 INFO:teuthology.orchestra.run.smithi064.stdout:5338 write 0x6f9a4a9 thru 0x6fa686b (0xc3c3 bytes) 2024-04-24T00:37:00.941 INFO:teuthology.orchestra.run.smithi064.stdout:5339 punch from 0x5fdd733 to 0x5fe6b4c, (0x9419 bytes) 2024-04-24T00:37:00.941 INFO:teuthology.orchestra.run.smithi064.stdout:5341 punch from 0x5b26978 to 0x5b36011, (0xf699 bytes) 2024-04-24T00:37:00.941 INFO:teuthology.orchestra.run.smithi064.stdout:5342 write 0x1f4f81b thru 0x1f5dce8 (0xe4ce bytes) 2024-04-24T00:37:00.955 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:00.950+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:00.963 INFO:teuthology.orchestra.run.smithi064.stdout:5343 read 0xae0a90d thru 0xae0c064 (0x1758 bytes) 2024-04-24T00:37:00.963 INFO:teuthology.orchestra.run.smithi064.stdout:5344 read 0x19847d2 thru 0x198f8e3 (0xb112 bytes) 2024-04-24T00:37:00.964 INFO:teuthology.orchestra.run.smithi064.stdout:5345 write 0xde754e2 thru 0xde7d9ee (0x850d bytes) 2024-04-24T00:37:00.981 INFO:teuthology.orchestra.run.smithi064.stdout:5346 read 0xa1e3b2d thru 0xa1e3b58 (0x2c bytes) 2024-04-24T00:37:00.996 INFO:teuthology.orchestra.run.smithi064.stdout:5347 trunc from 0xe596f28 to 0x7e6c0ec 2024-04-24T00:37:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:5352 read 0x101f619 thru 0x102dafd (0xe4e5 bytes) 2024-04-24T00:37:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:01.037 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:01.037 INFO:teuthology.orchestra.run.smithi064.stdout:5353 punch from 0x4286635 to 0x429375a, (0xd125 bytes) 2024-04-24T00:37:01.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:01.055 INFO:teuthology.orchestra.run.smithi064.stdout:5354 write 0x98b7800 thru 0x98be41c (0x6c1d bytes) 2024-04-24T00:37:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:5358 trunc from 0x98be41d to 0x3e3e70 2024-04-24T00:37:01.092 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:01.087+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:01.152 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:01.152 INFO:teuthology.orchestra.run.smithi064.stdout:5361 read 0x8c713 thru 0x90968 (0x4256 bytes) 2024-04-24T00:37:01.157 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:01.151+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:01.158 INFO:teuthology.orchestra.run.smithi064.stdout:5362 trunc from 0x3e3e70 to 0x2f1f3ab 2024-04-24T00:37:01.168 INFO:teuthology.orchestra.run.smithi064.stdout:5364 read 0x189c6b thru 0x193ebb (0xa251 bytes) 2024-04-24T00:37:01.188 INFO:teuthology.orchestra.run.smithi064.stdout:5365 writesame 0x7816863 thru 0x781fba3 data_size 0x17(0x9341 bytes) 2024-04-24T00:37:01.188 INFO:teuthology.orchestra.run.smithi064.stdout:5366 write 0x390f9d6 thru 0x3914fdb (0x5606 bytes) 2024-04-24T00:37:01.216 INFO:teuthology.orchestra.run.smithi064.stdout:5368 read 0x76af8af thru 0x76b564e (0x5da0 bytes) 2024-04-24T00:37:01.216 INFO:teuthology.orchestra.run.smithi064.stdout:5370 write 0x5adacab thru 0x5ae0715 (0x5a6b bytes) 2024-04-24T00:37:01.233 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:01.233 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:01.243 INFO:teuthology.orchestra.run.smithi064.stdout:5373 punch from 0x4e15a02 to 0x4e25764, (0xfd62 bytes) 2024-04-24T00:37:01.243 INFO:teuthology.orchestra.run.smithi064.stdout:5374 read 0x508d630 thru 0x508f8bc (0x228d bytes) 2024-04-24T00:37:01.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:01.258 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:01.254+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:01.264 INFO:teuthology.orchestra.run.smithi064.stdout:5377 write 0xd6733f6 thru 0xd677973 (0x457e bytes) 2024-04-24T00:37:01.292 INFO:teuthology.orchestra.run.smithi064.stdout:5378 read 0x65ab89c thru 0x65bae4c (0xf5b1 bytes) 2024-04-24T00:37:01.292 INFO:teuthology.orchestra.run.smithi064.stdout:5379 write 0xb106a6b thru 0xb10d368 (0x68fe bytes) 2024-04-24T00:37:01.316 INFO:teuthology.orchestra.run.smithi064.stdout:5380 write 0x48a9a99 thru 0x48b7bbe (0xe126 bytes) 2024-04-24T00:37:01.340 INFO:teuthology.orchestra.run.smithi064.stdout:5382 write 0x4ff5bbe thru 0x4ffd44f (0x7892 bytes) 2024-04-24T00:37:01.359 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:01.355+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:01.383 INFO:teuthology.orchestra.run.smithi064.stdout:5383 trunc from 0xd677974 to 0x4947c79 2024-04-24T00:37:01.389 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:01.389 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:01.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:01.427 INFO:teuthology.orchestra.run.smithi064.stdout:5384 write 0x623fb5 thru 0x62a03e (0x608a bytes) 2024-04-24T00:37:01.462 INFO:teuthology.orchestra.run.smithi064.stdout:5385 trunc from 0x4947c79 to 0x6ea1e6f 2024-04-24T00:37:01.462 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:01.459+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:01.471 INFO:teuthology.orchestra.run.smithi064.stdout:5388 write 0x47b2fb6 thru 0x47beb7b (0xbbc6 bytes) 2024-04-24T00:37:01.495 INFO:teuthology.orchestra.run.smithi064.stdout:5389 punch from 0x156dde0 to 0x1579c82, (0xbea2 bytes) 2024-04-24T00:37:01.496 INFO:teuthology.orchestra.run.smithi064.stdout:5392 write 0x2a1a187 thru 0x2a1ae76 (0xcf0 bytes) 2024-04-24T00:37:01.523 INFO:teuthology.orchestra.run.smithi064.stdout:5393 write 0xcd09fbe thru 0xcd19e2b (0xfe6e bytes) 2024-04-24T00:37:01.550 INFO:teuthology.orchestra.run.smithi064.stdout:5394 read 0x77796dc thru 0x77829fb (0x9320 bytes) 2024-04-24T00:37:01.550 INFO:teuthology.orchestra.run.smithi064.stdout:5395 punch from 0x9a2444a to 0x9a28c7d, (0x4833 bytes) 2024-04-24T00:37:01.550 INFO:teuthology.orchestra.run.smithi064.stdout:5396 read 0x286fb8 thru 0x289506 (0x254f bytes) 2024-04-24T00:37:01.551 INFO:teuthology.orchestra.run.smithi064.stdout:5397 read 0x44e2833 thru 0x44e79f0 (0x51be bytes) 2024-04-24T00:37:01.552 INFO:teuthology.orchestra.run.smithi064.stdout:5398 read 0x38c8ab1 thru 0x38cc58e (0x3ade bytes) 2024-04-24T00:37:01.554 INFO:teuthology.orchestra.run.smithi064.stdout:5401 write 0x6d4eff2 thru 0x6d5a094 (0xb0a3 bytes) 2024-04-24T00:37:01.561 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:01.558+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:01.564 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:01.564 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:01.577 INFO:teuthology.orchestra.run.smithi064.stdout:5402 read 0x7b130ba thru 0x7b200d9 (0xd020 bytes) 2024-04-24T00:37:01.577 INFO:teuthology.orchestra.run.smithi064.stdout:5403 punch from 0xa761173 to 0xa762801, (0x168e bytes) 2024-04-24T00:37:01.577 INFO:teuthology.orchestra.run.smithi064.stdout:5404 punch from 0x1bbcaee to 0x1bbdec7, (0x13d9 bytes) 2024-04-24T00:37:01.577 INFO:teuthology.orchestra.run.smithi064.stdout:5405 read 0x69f7e06 thru 0x6a057e0 (0xd9db bytes) 2024-04-24T00:37:01.577 INFO:teuthology.orchestra.run.smithi064.stdout:5406 punch from 0xa0427ac to 0xa044976, (0x21ca bytes) 2024-04-24T00:37:01.577 INFO:teuthology.orchestra.run.smithi064.stdout:5407 punch from 0x84f030 to 0x851ca8, (0x2c78 bytes) 2024-04-24T00:37:01.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:01.598 INFO:teuthology.orchestra.run.smithi064.stdout:5409 trunc from 0xcd19e2c to 0x639d6b 2024-04-24T00:37:01.662 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:01.658+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:01.677 INFO:teuthology.orchestra.run.smithi064.stdout:5410 writesame 0xaca504b thru 0xaca913c data_size 0x33(0x40f2 bytes) 2024-04-24T00:37:01.677 INFO:teuthology.orchestra.run.smithi064.stdout:5411 read 0xa4c3809 thru 0xa4c7689 (0x3e81 bytes) 2024-04-24T00:37:01.677 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:01.691 INFO:teuthology.orchestra.run.smithi064.stdout:5413 writesame 0xcaca548 thru 0xcad628c data_size 0x1f(0xbd45 bytes) 2024-04-24T00:37:01.692 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:01.692 INFO:teuthology.orchestra.run.smithi064.stdout:5417 write 0x5870839 thru 0x587c8c6 (0xc08e bytes) 2024-04-24T00:37:01.723 INFO:teuthology.orchestra.run.smithi064.stdout:5418 write 0x25299c thru 0x261c70 (0xf2d5 bytes) 2024-04-24T00:37:01.730 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:01.731 INFO:teuthology.orchestra.run.smithi064.stdout:5422 trunc from 0xcad628d to 0xd0af804 2024-04-24T00:37:01.740 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:01.740 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:01.742 INFO:teuthology.orchestra.run.smithi064.stdout:5424 read 0x8f634f8 thru 0x8f6513c (0x1c45 bytes) 2024-04-24T00:37:01.742 INFO:teuthology.orchestra.run.smithi064.stdout:5425 read 0x718f491 thru 0x719c4d5 (0xd045 bytes) 2024-04-24T00:37:01.757 INFO:teuthology.orchestra.run.smithi064.stdout:5426 write 0xebc340c thru 0xebd0363 (0xcf58 bytes) 2024-04-24T00:37:01.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:01.763 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:01.758+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:01.787 INFO:teuthology.orchestra.run.smithi064.stdout:5429 writesame 0x5334caa thru 0x5344c78 data_size 0x53(0xffcf bytes) 2024-04-24T00:37:01.816 INFO:teuthology.orchestra.run.smithi064.stdout:5430 write 0xebaec5 thru 0xebb6c9 (0x805 bytes) 2024-04-24T00:37:01.825 INFO:teuthology.orchestra.run.smithi064.stdout:5431 write 0x3be4441 thru 0x3bf3869 (0xf429 bytes) 2024-04-24T00:37:01.852 INFO:teuthology.orchestra.run.smithi064.stdout:5432 read 0x3456db5 thru 0x3462ca2 (0xbeee bytes) 2024-04-24T00:37:01.852 INFO:teuthology.orchestra.run.smithi064.stdout:5433 read 0x1155c5c thru 0x115f8a7 (0x9c4c bytes) 2024-04-24T00:37:01.853 INFO:teuthology.orchestra.run.smithi064.stdout:5436 punch from 0x97337ff to 0x973ed4a, (0xb54b bytes) 2024-04-24T00:37:01.853 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:01.853 INFO:teuthology.orchestra.run.smithi064.stdout:5439 write 0x5930dd5 thru 0x59368c3 (0x5aef bytes) 2024-04-24T00:37:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:5440 punch from 0x9a8f0d6 to 0x9a94fe7, (0x5f11 bytes) 2024-04-24T00:37:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:5441 write 0xa91ff12 thru 0xa926e1c (0x6f0b bytes) 2024-04-24T00:37:01.863 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:01.859+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:01.884 INFO:teuthology.orchestra.run.smithi064.stdout:5442 write 0xb97a34e thru 0xb9877d9 (0xd48c bytes) 2024-04-24T00:37:01.891 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:01.892 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:01.892 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:01.905 INFO:teuthology.orchestra.run.smithi064.stdout:5443 write 0x14fca45 thru 0x1506b2e (0xa0ea bytes) 2024-04-24T00:37:01.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:01.928 INFO:teuthology.orchestra.run.smithi064.stdout:5444 punch from 0x6506488 to 0x650a00b, (0x3b83 bytes) 2024-04-24T00:37:01.949 INFO:teuthology.orchestra.run.smithi064.stdout:5445 trunc from 0xebd0364 to 0x2b5a386 2024-04-24T00:37:01.964 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:01.959+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:01.984 INFO:teuthology.orchestra.run.smithi064.stdout:5446 punch from 0xd13ab9 to 0xd14bfc, (0x1143 bytes) 2024-04-24T00:37:01.984 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:01.984 INFO:teuthology.orchestra.run.smithi064.stdout:5448 read 0x2993f48 thru 0x299c367 (0x8420 bytes) 2024-04-24T00:37:01.985 INFO:teuthology.orchestra.run.smithi064.stdout:5449 write 0xf6643d thru 0xf66502 (0xc6 bytes) 2024-04-24T00:37:01.999 INFO:teuthology.orchestra.run.smithi064.stdout:5450 trunc from 0x2b5a386 to 0x9668803 2024-04-24T00:37:02.010 INFO:teuthology.orchestra.run.smithi064.stdout:5451 trunc from 0x9668803 to 0x9f57496 2024-04-24T00:37:02.022 INFO:teuthology.orchestra.run.smithi064.stdout:5453 writesame 0x7bfc06e thru 0x7c0aad6 data_size 0x53(0xea69 bytes) 2024-04-24T00:37:02.056 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:02.056 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:02.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:02.058 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:02.063 INFO:teuthology.orchestra.run.smithi064.stdout:5457 write 0x4d8e5ac thru 0x4d91d51 (0x37a6 bytes) 2024-04-24T00:37:02.065 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:02.059+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:02.074 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:02.075 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:02.075 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:02.082 INFO:teuthology.orchestra.run.smithi064.stdout:5458 read 0x594a087 thru 0x5953dff (0x9d79 bytes) 2024-04-24T00:37:02.083 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:02.083 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:02.083 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-24T00:37:02.084 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:37:02.084 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:37:02.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-04-24T00:37:02.093 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:02.093 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:02.096 INFO:teuthology.orchestra.run.smithi064.stdout:5459 writesame 0xdc11ca1 thru 0xdc17d43 data_size 0xd(0x60a3 bytes) 2024-04-24T00:37:02.096 INFO:teuthology.orchestra.run.smithi064.stdout:5461 read 0x28a71fd thru 0x28aba0f (0x4813 bytes) 2024-04-24T00:37:02.096 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:02.096 INFO:teuthology.orchestra.run.smithi064.stdout:5464 read 0x48e0d37 thru 0x48ef3f7 (0xe6c1 bytes) 2024-04-24T00:37:02.096 INFO:teuthology.orchestra.run.smithi064.stdout:5465 punch from 0xd269b87 to 0xd278fd1, (0xf44a bytes) 2024-04-24T00:37:02.096 INFO:teuthology.orchestra.run.smithi064.stdout:5466 write 0x3599da0 thru 0x35a633f (0xc5a0 bytes) 2024-04-24T00:37:02.112 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:02.112 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:02.113 INFO:teuthology.orchestra.run.smithi064.stdout:5469 trunc from 0xdc17d44 to 0xbb6079b 2024-04-24T00:37:02.122 INFO:teuthology.orchestra.run.smithi064.stdout:5470 punch from 0x9ad691e to 0x9ae0e67, (0xa549 bytes) 2024-04-24T00:37:02.122 INFO:teuthology.orchestra.run.smithi064.stdout:5472 write 0x7ea77e1 thru 0x7ea9e8d (0x26ad bytes) 2024-04-24T00:37:02.127 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:02.127 INFO:teuthology.orchestra.run.smithi064.stdout:5474 write 0x6a022c4 thru 0x6a11c70 (0xf9ad bytes) 2024-04-24T00:37:02.132 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:02.132 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:02.152 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:02.161 INFO:teuthology.orchestra.run.smithi064.stdout:5477 write 0xee1135b thru 0xee1771d (0x63c3 bytes) 2024-04-24T00:37:02.166 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:02.162+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:02.173 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:02.173 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:02.179 INFO:teuthology.orchestra.run.smithi064.stdout:5478 punch from 0x1cac036 to 0x1cb24ec, (0x64b6 bytes) 2024-04-24T00:37:02.179 INFO:teuthology.orchestra.run.smithi064.stdout:5479 write 0xbc35396 thru 0xbc388ea (0x3555 bytes) 2024-04-24T00:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:02.193 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:02.195 INFO:teuthology.orchestra.run.smithi064.stdout:5480 writesame 0x841d70c thru 0x8428b4d data_size 0x3(0xb442 bytes) 2024-04-24T00:37:02.217 INFO:teuthology.orchestra.run.smithi064.stdout:5481 read 0xd619df5 thru 0xd6247ce (0xa9da bytes) 2024-04-24T00:37:02.217 INFO:teuthology.orchestra.run.smithi064.stdout:5483 read 0x10ad148 thru 0x10b54eb (0x83a4 bytes) 2024-04-24T00:37:02.218 INFO:teuthology.orchestra.run.smithi064.stdout:5484 writesame 0x972ecd5 thru 0x973c50c data_size 0xb(0xd838 bytes) 2024-04-24T00:37:02.231 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:02.231 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:02.231 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:02.231 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:02.231 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:02.238 INFO:teuthology.orchestra.run.smithi064.stdout:5485 writesame 0x50b141b thru 0x50b8734 data_size 0x1(0x731a bytes) 2024-04-24T00:37:02.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:02.262 INFO:teuthology.orchestra.run.smithi064.stdout:5487 read 0x3b426cb thru 0x3b481f3 (0x5b29 bytes) 2024-04-24T00:37:02.262 INFO:teuthology.orchestra.run.smithi064.stdout:5488 write 0xcb93b51 thru 0xcb9e9d5 (0xae85 bytes) 2024-04-24T00:37:02.269 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:02.269 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T00:37:02.269 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:02.269 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:02.266+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:02.277 INFO:teuthology.orchestra.run.smithi064.stdout:5489 writesame 0xb73d151 thru 0xb74a0a8 data_size 0x14(0xcf58 bytes) 2024-04-24T00:37:02.296 INFO:teuthology.orchestra.run.smithi064.stdout:5490 read 0x9a45009 thru 0x9a50300 (0xb2f8 bytes) 2024-04-24T00:37:02.296 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:02.296 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:02.296 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout:5497 trunc from 0xee1771e to 0xa9f7e0 2024-04-24T00:37:02.368 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:02.363+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.190850063, 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:37:02.407 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:37:02.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:02.412 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 154.76906436900001, 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:02.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:37:02.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:37:02.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:37:02.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 93.626936803000007, 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:37:02.417 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:37:02.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:37:02.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:37:02.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 88.489165021000005, 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.423 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:02.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:37:02.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:37:02.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:37:02.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:02.428 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 88.451682703000003, 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:02.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:37:02.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:84137 3.1 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head [write 0~4014080 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 88.225091837999997, 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076296485999999997, 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 84137 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180083+0000", 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.180085+0000", 2024-04-24T00:37:02.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4190000000000001e-06 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189051+0000", 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089667529999999992 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189072+0000", 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0271e-05 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189084+0000", 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2486999999999999e-05 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189188+0000", 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103769 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189414+0000", 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022651600000000001 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.189645+0000", 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000230647 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.219745+0000", 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030100077999999999 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256320+0000", 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036575023999999998 2024-04-24T00:37:02.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256356+0000", 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5441999999999997e-05 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.256380+0000", 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.408e-05 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.439 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.789336839000001, 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:37:02.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:37:02.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.779478803000003, 2024-04-24T00:37:02.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:37:02.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:37:02.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 84.759129195, 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.451 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.456 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 73.725083859999998, 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:02.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 73.722774024000003, 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.461 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.466 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89929 3.13 3:cad47bd2:::rbd_data.115e56aa02fc.0000000000000002:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 73.714205225000001, 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076349195999999994, 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89929 2024-04-24T00:37:02.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690970+0000", 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690973+0000", 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704677+0000", 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013703768 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704722+0000", 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5275999999999998e-05 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.704746+0000", 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4360000000000001e-05 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.706827+0000", 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002080658 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.707712+0000", 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088515700000000002 2024-04-24T00:37:02.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.708672+0000", 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00096006300000000004 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.766798+0000", 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058125783 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767233+0000", 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043520800000000002 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767277+0000", 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3534e-05 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.767319+0000", 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2358999999999999e-05 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.472 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.775056132000003, 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:37:02.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:37:02.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.885666135999998, 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.479 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.484 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.842190338999998, 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:37:02.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:37:02.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.868217037000001, 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:37:02.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.494 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94354 3.14 e146/106 ondisk, result = 0)", 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.849075134000003, 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076944002999999997, 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556100+0000", 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556102+0000", 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556111+0000", 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9770000000000006e-06 2024-04-24T00:37:02.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556146+0000", 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5063e-05 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.556165+0000", 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9309000000000001e-05 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632792+0000", 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076626386000000005 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632810+0000", 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.802e-05 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633044+0000", 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000233978 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.498 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.889787581, 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:02.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:37:02.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.855785159, 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.504 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:37:02.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:126987 3.11 3:8f8290b4:::rbd_data.115ee44a9909.0000000000000003:46 [read 522240~2048 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.691331461000001, 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077109041000000003, 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:02.508 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 126987 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:37:02.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713844+0000", 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2500000000000005e-07 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713852+0000", 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:02.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713859+0000", 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0720000000000002e-06 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.713864+0000", 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2900000000000002e-06 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790095+0000", 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076231499999999994 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790170+0000", 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4986000000000003e-05 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.790953+0000", 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00078209500000000001 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:02.512 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:02.514 INFO:teuthology.orchestra.run.smithi064.stdout:5498 writesame 0xc166242 thru 0xc173dee data_size 0x1(0xdbad bytes) 2024-04-24T00:37:02.514 INFO:teuthology.orchestra.run.smithi064.stdout:5499 punch from 0xafa25f9 to 0xafad8a7, (0xb2ae bytes) 2024-04-24T00:37:02.514 INFO:teuthology.orchestra.run.smithi064.stdout:5500 write 0x3b878a8 thru 0x3b90605 (0x8d5e bytes) 2024-04-24T00:37:02.514 INFO:teuthology.orchestra.run.smithi064.stdout:5502 write 0xdef5a1 thru 0xdf15c9 (0x2029 bytes) 2024-04-24T00:37:02.514 INFO:teuthology.orchestra.run.smithi064.stdout:5504 writesame 0x6044171 thru 0x6051940 data_size 0x18(0xd7d0 bytes) 2024-04-24T00:37:02.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:02.515 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:02.510+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:02.520 INFO:teuthology.orchestra.run.smithi064.stdout:5505 read 0x3566627 thru 0x35686f0 (0x20ca bytes) 2024-04-24T00:37:02.521 INFO:teuthology.orchestra.run.smithi064.stdout:5508 punch from 0x1afe2c0 to 0x1aff004, (0xd44 bytes) 2024-04-24T00:37:02.521 INFO:teuthology.orchestra.run.smithi064.stdout:5510 writesame 0x9395a7 thru 0x9489e1 data_size 0x3(0xf43b bytes) 2024-04-24T00:37:02.554 INFO:teuthology.orchestra.run.smithi064.stdout:5511 write 0x6320c85 thru 0x6330aed (0xfe69 bytes) 2024-04-24T00:37:02.592 INFO:teuthology.orchestra.run.smithi064.stdout:5513 trunc from 0xc173def to 0x18e90b3 2024-04-24T00:37:02.615 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:02.611+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:02.621 INFO:teuthology.orchestra.run.smithi064.stdout:5514 read 0x16f7dbc thru 0x1706899 (0xeade bytes) 2024-04-24T00:37:02.665 INFO:teuthology.orchestra.run.smithi064.stdout:5515 write 0xbd28df1 thru 0xbd2a23a (0x144a bytes) 2024-04-24T00:37:02.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:37:02.683 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:02.683 INFO:teuthology.orchestra.run.smithi064.stdout:5517 punch from 0x41ce4e3 to 0x41d9e5d, (0xb97a bytes) 2024-04-24T00:37:02.683 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:02.683 INFO:teuthology.orchestra.run.smithi064.stdout:5519 punch from 0x5629412 to 0x5631a6b, (0x8659 bytes) 2024-04-24T00:37:02.683 INFO:teuthology.orchestra.run.smithi064.stdout:5521 writesame 0x806d858 thru 0x807a6c9 data_size 0x23(0xce72 bytes) 2024-04-24T00:37:02.693 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:02.693 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:02.693 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:02.703 INFO:teuthology.orchestra.run.smithi064.stdout:5522 write 0xbd903e thru 0xbe6638 (0xd5fb bytes) 2024-04-24T00:37:02.708 INFO:teuthology.orchestra.run.smithi064.stdout:5523 read 0x16b825c thru 0x16c1e58 (0x9bfd bytes) 2024-04-24T00:37:02.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:02.720 INFO:teuthology.orchestra.run.smithi064.stdout:5524 trunc from 0xbd2a23b to 0x766b11f 2024-04-24T00:37:02.743 INFO:teuthology.orchestra.run.smithi064.stdout:5525 write 0x60f150d thru 0x60f5986 (0x447a bytes) 2024-04-24T00:37:02.757 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:02.754+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:02.761 INFO:teuthology.orchestra.run.smithi064.stdout:5526 write 0x9a9fd thru 0x9f2ed (0x48f1 bytes) 2024-04-24T00:37:02.770 INFO:teuthology.orchestra.run.smithi064.stdout:5527 write 0x4bce865 thru 0x4bdaf38 (0xc6d4 bytes) 2024-04-24T00:37:02.793 INFO:teuthology.orchestra.run.smithi064.stdout:5529 read 0x3dd6e02 thru 0x3ddb363 (0x4562 bytes) 2024-04-24T00:37:02.793 INFO:teuthology.orchestra.run.smithi064.stdout:5531 punch from 0x7372376 to 0x737ea39, (0xc6c3 bytes) 2024-04-24T00:37:02.793 INFO:teuthology.orchestra.run.smithi064.stdout:5532 clone 37 order 20 su 262144 sc 4 2024-04-24T00:37:02.816 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:02.810+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:02.837 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:02.837 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:02.837 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:02.837 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:02.837 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:02.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:02.937 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:02.934+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:03.017 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.30542063799999, 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.018 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:37:03.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.022 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.023 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.30532271300001, 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.024 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:37:03.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.031 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:03.031 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:03.031 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.94594318099999, 2024-04-24T00:37:03.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:37:03.031 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:37:03.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:37:03.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.037 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.53519905500001, 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.038 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.509930449, 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.656726729, 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:37:03.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:37:03.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.055 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.62412161500001, 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:03.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:37:03.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.219546921, 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:37:03.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:37:03.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:37:03.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:37:03.065 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.16170669100001, 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:37:03.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.320144342000006, 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:37:03.070 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:37:03.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:37:03.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:37:03.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.341396791000001, 2024-04-24T00:37:03.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:37:03.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.311754737999998, 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:37:03.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 57.165239327000002, 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:37:03.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.376466768999997, 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:37:03.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:37:03.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:37:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.097 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.353323355999997, 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:37:03.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:37:03.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.480014607999998, 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:37:03.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:37:03.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:37:03.108 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.427332331000002, 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:37:03.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127274 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.346247472000002, 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071400004000000003, 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.113 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127274 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667542+0000", 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667545+0000", 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9059999999999998e-06 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667553+0000", 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4910000000000005e-06 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667560+0000", 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5339999999999996e-06 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738552+0000", 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070991992000000004 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738615+0000", 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3559000000000006e-05 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738941+0000", 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032590199999999998 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127275 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.346219758, 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071850050999999998, 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127275 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.118 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:03.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4000000000000002e-07 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667572+0000", 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9780000000000001e-06 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667581+0000", 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7080000000000003e-06 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667587+0000", 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738971+0000", 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071383263000000002 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739053+0000", 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2591999999999998e-05 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739419+0000", 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036566600000000001 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:128046 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.284791251000001, 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072176436999999996, 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 128046 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728998+0000", 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8299999999999998e-07 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729000+0000", 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2479999999999999e-06 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729006+0000", 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9569999999999999e-06 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729010+0000", 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4730000000000002e-06 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800122+0000", 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071111555000000007 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800239+0000", 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117401 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.801174+0000", 2024-04-24T00:37:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093442000000000004 2024-04-24T00:37:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.126 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.126 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:03.126 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:03.014+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:03.127 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:03.169 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:03.166+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:03.230 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:03.226+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:03.296 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:03.297 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:03.297 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:03.297 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:03.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:03.331 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:03.327+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:03.432 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:03.426+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:03.478 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:03.478 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:03.478 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:03.478 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:03.478 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:03.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:03.533 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:03.530+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:03.634 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:03.631+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:03.656 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 165.12030704399999, 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.657 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:03.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:37:03.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:37:03.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.662 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 165.08626999099999, 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:37:03.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.666 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.971155995, 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:37:03.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:37:03.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:37:03.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.943706034, 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:37:03.676 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.023634201999997, 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.017362943999998, 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.681 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:37:03.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:37:03.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.005908357999999, 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:37:03.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:37:03.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:03.691 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 85.951455813999999, 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.695 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105171 3.1e 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head [zero 0~262144,write 262144~131072 [fadvise_sequential+fadvise_nocache],zero 393216~131072,write 524288~917504 [fadvise_sequential+fadvise_nocache],zero 1441792~393216,write 1835008~131072 [fadvise_sequential+fadvise_nocache],write 2097152~917504 [fadvise_sequential+fadvise_nocache],zero 3014656~393216,write 3407872~131072 [fadvise_sequential+fadvise_nocache],zero 3538944~131072,write 3670016~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.961705322, 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061400402, 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105171 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690858+0000", 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5059999999999998e-06 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697752+0000", 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006893766 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697795+0000", 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2589000000000002e-05 2024-04-24T00:37:03.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697810+0000", 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5620999999999998e-05 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697936+0000", 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012529199999999999 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698150+0000", 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021430200000000001 2024-04-24T00:37:03.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698445+0000", 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029468099999999998 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736712+0000", 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038267781000000001 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752153+0000", 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015441043999999999 2024-04-24T00:37:03.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752204+0000", 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752256+0000", 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2286e-05 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.701 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127048 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 378880~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.002512675999998, 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768986, 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127048 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.92e-07 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650053+0000", 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.698e-06 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650065+0000", 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1454000000000001e-05 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650073+0000", 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8920000000000006e-06 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705378+0000", 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055304820999999997 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705456+0000", 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8009999999999993e-05 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705818+0000", 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036131899999999997 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127049 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 382976~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.002475504, 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056235535000000003, 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127049 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650090+0000", 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650101+0000", 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0822000000000001e-05 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650110+0000", 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705851+0000", 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055741377000000002 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705916+0000", 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5160000000000006e-05 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706321+0000", 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040486500000000002 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127050 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 387072~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.002440043, 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056659432000000003, 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127050 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650122+0000", 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650125+0000", 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.366e-06 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650136+0000", 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0624e-05 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650144+0000", 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4360000000000002e-06 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706354+0000", 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056210006999999999 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706437+0000", 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2515000000000005e-05 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706781+0000", 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034373599999999998 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.713 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127051 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.002406091000001, 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057055097999999999, 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127051 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650156+0000", 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1399999999999996e-07 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650159+0000", 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3100000000000001e-06 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650170+0000", 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.058e-05 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650178+0000", 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5660000000000007e-06 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706827+0000", 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056648506000000001 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706903+0000", 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6493000000000001e-05 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.707210+0000", 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306929 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.717 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127335 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.988090475, 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055806472000000003, 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.718 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127335 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5200000000000002e-07 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664474+0000", 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6400000000000001e-06 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664481+0000", 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.002e-06 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664493+0000", 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2311e-05 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719879+0000", 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055386187000000003 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719956+0000", 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6196e-05 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720277+0000", 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032168399999999998 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127336 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.988057962999999, 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056212985, 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.722 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127336 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664504+0000", 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5400000000000001e-07 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664513+0000", 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.132e-06 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664520+0000", 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3849999999999996e-06 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664525+0000", 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0180000000000001e-06 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720371+0000", 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055845976999999998 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720430+0000", 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8964e-05 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720716+0000", 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028595500000000001 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.726 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127337 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.98802659, 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056616423999999999, 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127337 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2099999999999997e-07 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664538+0000", 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0410000000000002e-06 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664546+0000", 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3710000000000002e-06 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664551+0000", 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0889999999999999e-06 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720759+0000", 2024-04-24T00:37:03.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056208658000000002 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720811+0000", 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1564999999999998e-05 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721151+0000", 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034017900000000002 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127338 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.98800245, 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057296623999999997, 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127338 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.58e-07 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664562+0000", 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.345e-06 2024-04-24T00:37:03.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664568+0000", 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664573+0000", 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721197+0000", 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056624102000000003 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721296+0000", 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9001999999999998e-05 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721855+0000", 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055934300000000001 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127339 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.987981904000002, 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057818889999999998, 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127339 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664580+0000", 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5299999999999999e-07 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664582+0000", 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1399999999999998e-06 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664588+0000", 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1959999999999996e-06 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664593+0000", 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.656e-06 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721894+0000", 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057301501999999997 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721991+0000", 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6330999999999996e-05 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.722398+0000", 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040761200000000001 2024-04-24T00:37:03.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.276607178999999, 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:37:03.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:37:03.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.255715632999999, 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:03.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:37:03.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:37:03.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:37:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:03.748 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:03.748 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:03.750 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:37:03.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:03.750 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:03.746+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:03.849 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:03.847+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:03.924 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:03.924 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:03.924 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:03.924 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:03.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:03.985 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:03.982+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:04.051 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:04.046+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:04.061 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:04.062 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:04.062 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:04.062 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:04.062 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:04.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:04.153 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:04.147+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:04.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:04.245 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:04.245 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 203.31558788300001, 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:37:04.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:37:04.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 170.38362219499999, 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.252 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:04.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 170.38237344699999, 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:37:04.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:37:04.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:04.263 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 161.22474934300001, 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:04.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:37:04.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:37:04.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 161.21125480000001, 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.269 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:37:04.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.94947368499999, 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.273 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:37:04.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 147.69588754200001, 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:37:04.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:37:04.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 147.668213678, 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.285 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:37:04.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.289 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.174190858, 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:37:04.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:37:04.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.295 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.16518202399999, 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.299 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.66348307600001, 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:37:04.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:37:04.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.66342073, 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:37:04.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:37:04.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.66291105099999, 2024-04-24T00:37:04.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:37:04.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:37:04.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.66279998900001, 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:37:04.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:37:04.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:37:04.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.662734723, 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.317 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.662674657, 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.321 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:37:04.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.66261796400001, 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:37:04.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:37:04.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:37:04.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.66254863100001, 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.662458012, 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.334 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:04.338 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.662414599, 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:04.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:04.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.342 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.343 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.343 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:04.343 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:04.343 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:04.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:04.345 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:04.339+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:04.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:04.374 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:04.391 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:04.391 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:04.409 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:04.426 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:04.426 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:04.445 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:04.442+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:04.447 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:04.463 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:04.463 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:04.468 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:04.468 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:04.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:04.489 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:04.489 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:04.510 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:04.510 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:04.573 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:04.571+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:04.622 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:04.622 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:04.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:04.682 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:04.679+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:04.747 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:04.743+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:04.829 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:04.829 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:04.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:04.848 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:04.842+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:04.950 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:04.946+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:05.018 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:05.018 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:05.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:05.077 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:05.075+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:05.152 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:05.146+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:05.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:05.253 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:05.250+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:05.341 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:05.341 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:05.353 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:05.350+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:05.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:05.454 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:05.450+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:05.482 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:05.482 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:05.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:05.555 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:05.550+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:05.616 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:05.616 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:05.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:05.656 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:05.650+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:05.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:37:05.757 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:05.754+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:05.758 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:05.759 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:05.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:05.813 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #35, image image_client.0-clone35 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent36 2024-04-24T00:37:05.858 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:05.854+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:05.949 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:05.949 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:05.959 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:05.955+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:05.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:06.060 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:06.054+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:06.062 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.069 INFO:teuthology.orchestra.run.smithi064.stdout:5534 trunc from 0x766b11f to 0xdcd0308 2024-04-24T00:37:06.085 INFO:teuthology.orchestra.run.smithi064.stdout:5535 punch from 0x145a28b to 0x1460ccc, (0x6a41 bytes) 2024-04-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi064.stdout:5537 writesame 0xddb7b5e thru 0xddbb26d data_size 0x8(0x3710 bytes) 2024-04-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi064.stdout:5538 write 0x90fe0d8 thru 0x910784f (0x9778 bytes) 2024-04-24T00:37:06.102 INFO:teuthology.orchestra.run.smithi064.stdout:5539 write 0xc1f5d8 thru 0xc2712f (0x7b58 bytes) 2024-04-24T00:37:06.111 INFO:teuthology.orchestra.run.smithi064.stdout:5540 read 0xbcec17a thru 0xbcf4ea8 (0x8d2f bytes) 2024-04-24T00:37:06.111 INFO:teuthology.orchestra.run.smithi064.stdout:5542 punch from 0x8bf6253 to 0x8bfa6cc, (0x4479 bytes) 2024-04-24T00:37:06.111 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.111 INFO:teuthology.orchestra.run.smithi064.stdout:5545 read 0x86883c6 thru 0x868c6c2 (0x42fd bytes) 2024-04-24T00:37:06.111 INFO:teuthology.orchestra.run.smithi064.stdout:5548 write 0x722202b thru 0x72269f4 (0x49ca bytes) 2024-04-24T00:37:06.119 INFO:teuthology.orchestra.run.smithi064.stdout:5549 punch from 0x52d8c26 to 0x52dc56d, (0x3947 bytes) 2024-04-24T00:37:06.119 INFO:teuthology.orchestra.run.smithi064.stdout:5550 write 0x92a6de8 thru 0x92a86c5 (0x18de bytes) 2024-04-24T00:37:06.130 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:06.130 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:06.135 INFO:teuthology.orchestra.run.smithi064.stdout:5553 writesame 0xe849fcd thru 0xe851a56 data_size 0x5(0x7a8a bytes) 2024-04-24T00:37:06.135 INFO:teuthology.orchestra.run.smithi064.stdout:5554 writesame 0xe4b40d1 thru 0xe4b8711 data_size 0x37(0x4641 bytes) 2024-04-24T00:37:06.141 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.141 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:06.145 INFO:teuthology.orchestra.run.smithi064.stdout:5558 trunc from 0xe851a57 to 0xb6ee940 2024-04-24T00:37:06.163 INFO:teuthology.orchestra.run.smithi064.stdout:5559 read 0x7a9366 thru 0x7b5321 (0xbfbc bytes) 2024-04-24T00:37:06.164 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:06.159+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:06.164 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.171 INFO:teuthology.orchestra.run.smithi064.stdout:5562 trunc from 0xb6ee940 to 0x500fc28 2024-04-24T00:37:06.209 INFO:teuthology.orchestra.run.smithi064.stdout:5565 write 0x684cc3b thru 0x685c9a2 (0xfd68 bytes) 2024-04-24T00:37:06.218 INFO:teuthology.orchestra.run.smithi064.stdout:5567 read 0x163bd70 thru 0x164b34e (0xf5df bytes) 2024-04-24T00:37:06.218 INFO:teuthology.orchestra.run.smithi064.stdout:5568 write 0x1541830 thru 0x15503f8 (0xebc9 bytes) 2024-04-24T00:37:06.253 INFO:teuthology.orchestra.run.smithi064.stdout:5570 write 0x8931c00 thru 0x894012c (0xe52d bytes) 2024-04-24T00:37:06.261 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:06.255+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:06.262 INFO:teuthology.orchestra.run.smithi064.stdout:5571 read 0xf28834 thru 0xf2f88c (0x7059 bytes) 2024-04-24T00:37:06.263 INFO:teuthology.orchestra.run.smithi064.stdout:5574 read 0x3332ada thru 0x3333fe2 (0x1509 bytes) 2024-04-24T00:37:06.263 INFO:teuthology.orchestra.run.smithi064.stdout:5575 write 0xaa98e7 thru 0xab58d4 (0xbfee bytes) 2024-04-24T00:37:06.264 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:06.264 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:06.273 INFO:teuthology.orchestra.run.smithi064.stdout:5576 read 0x337f860 thru 0x338a650 (0xadf1 bytes) 2024-04-24T00:37:06.273 INFO:teuthology.orchestra.run.smithi064.stdout:5578 read 0x4127d97 thru 0x412fd59 (0x7fc3 bytes) 2024-04-24T00:37:06.276 INFO:teuthology.orchestra.run.smithi064.stdout:5579 trunc from 0x894012d to 0x6a869f8 2024-04-24T00:37:06.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:06.298 INFO:teuthology.orchestra.run.smithi064.stdout:5580 write 0x3992618 thru 0x3996c11 (0x45fa bytes) 2024-04-24T00:37:06.303 INFO:teuthology.orchestra.run.smithi064.stdout:5581 punch from 0x513ce5a to 0x51449ab, (0x7b51 bytes) 2024-04-24T00:37:06.310 INFO:teuthology.orchestra.run.smithi064.stdout:5583 trunc from 0x6a869f8 to 0x9e4d695 2024-04-24T00:37:06.320 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.320 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.320 INFO:teuthology.orchestra.run.smithi064.stdout:5589 write 0x4bc4e8 thru 0x4c6dee (0xa907 bytes) 2024-04-24T00:37:06.323 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.335 INFO:teuthology.orchestra.run.smithi064.stdout:5592 writesame 0xb518b1f thru 0xb5218a4 data_size 0x5(0x8d86 bytes) 2024-04-24T00:37:06.337 INFO:teuthology.orchestra.run.smithi064.stdout:5594 trunc from 0xb5218a5 to 0xca50a30 2024-04-24T00:37:06.349 INFO:teuthology.orchestra.run.smithi064.stdout:5599 write 0x323e49 thru 0x32b981 (0x7b39 bytes) 2024-04-24T00:37:06.356 INFO:teuthology.orchestra.run.smithi064.stdout:5600 read 0xe3416 thru 0xe5a7a (0x2665 bytes) 2024-04-24T00:37:06.358 INFO:teuthology.orchestra.run.smithi064.stdout:5601 read 0x106fd46 thru 0x107f5c8 (0xf883 bytes) 2024-04-24T00:37:06.362 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:06.358+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:06.367 INFO:teuthology.orchestra.run.smithi064.stdout:5604 trunc from 0xca50a30 to 0x81acc6e 2024-04-24T00:37:06.396 INFO:teuthology.orchestra.run.smithi064.stdout:5605 write 0x8515f1e thru 0x8517a98 (0x1b7b bytes) 2024-04-24T00:37:06.415 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:06.415 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:06.428 INFO:teuthology.orchestra.run.smithi064.stdout:5607 writesame 0xbdad11f thru 0xbdb2f93 data_size 0x1(0x5e75 bytes) 2024-04-24T00:37:06.428 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.428 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.428 INFO:teuthology.orchestra.run.smithi064.stdout:5610 punch from 0x80c98fc to 0x80d2e42, (0x9546 bytes) 2024-04-24T00:37:06.428 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.429 INFO:teuthology.orchestra.run.smithi064.stdout:5612 read 0x211ebb3 thru 0x212e44a (0xf898 bytes) 2024-04-24T00:37:06.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:06.453 INFO:teuthology.orchestra.run.smithi064.stdout:5617 trunc from 0xbdb2f94 to 0x12a7f07 2024-04-24T00:37:06.473 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:06.470+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:06.503 INFO:teuthology.orchestra.run.smithi064.stdout:5618 writesame 0x31f1312 thru 0x31f4f2a data_size 0x11(0x3c19 bytes) 2024-04-24T00:37:06.503 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:06.504 INFO:teuthology.orchestra.run.smithi064.stdout:5621 read 0x27a554f thru 0x27b236d (0xce1f bytes) 2024-04-24T00:37:06.504 INFO:teuthology.orchestra.run.smithi064.stdout:5623 punch from 0xa173f5 to 0xa21465, (0xa070 bytes) 2024-04-24T00:37:06.504 INFO:teuthology.orchestra.run.smithi064.stdout:5624 write 0x1fa174c thru 0x1fa86d7 (0x6f8c bytes) 2024-04-24T00:37:06.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:06.538 INFO:teuthology.orchestra.run.smithi064.stdout:5625 write 0x780b3e9 thru 0x781ab17 (0xf72f bytes) 2024-04-24T00:37:06.545 INFO:teuthology.orchestra.run.smithi064.stdout:5627 read 0x10e064e thru 0x10efa06 (0xf3b9 bytes) 2024-04-24T00:37:06.564 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:06.559+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:06.564 INFO:teuthology.orchestra.run.smithi064.stdout:5629 trunc from 0x781ab18 to 0xe728788 2024-04-24T00:37:06.592 INFO:teuthology.orchestra.run.smithi064.stdout:5631 trunc from 0xe728788 to 0x6e5b745 2024-04-24T00:37:06.596 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:06.596 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:06.613 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:06.643 INFO:teuthology.orchestra.run.smithi064.stdout:5633 write 0xe4333db thru 0xe435937 (0x255d bytes) 2024-04-24T00:37:06.650 INFO:teuthology.orchestra.run.smithi064.stdout:5634 punch from 0x7234e60 to 0x723cba1, (0x7d41 bytes) 2024-04-24T00:37:06.650 INFO:teuthology.orchestra.run.smithi064.stdout:5635 punch from 0x16ddf0e to 0x16e2ac7, (0x4bb9 bytes) 2024-04-24T00:37:06.661 INFO:teuthology.orchestra.run.smithi064.stdout:5636 trunc from 0xe435938 to 0xa11c127 2024-04-24T00:37:06.664 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:06.659+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:06.673 INFO:teuthology.orchestra.run.smithi064.stdout:5637 read 0x4d7d9e6 thru 0x4d87973 (0x9f8e bytes) 2024-04-24T00:37:06.673 INFO:teuthology.orchestra.run.smithi064.stdout:5638 punch from 0x5110862 to 0x511db5e, (0xd2fc bytes) 2024-04-24T00:37:06.673 INFO:teuthology.orchestra.run.smithi064.stdout:5640 punch from 0x6f5ca9e to 0x6f67904, (0xae66 bytes) 2024-04-24T00:37:06.674 INFO:teuthology.orchestra.run.smithi064.stdout:5641 punch from 0x7c1bc95 to 0x7c29f23, (0xe28e bytes) 2024-04-24T00:37:06.674 INFO:teuthology.orchestra.run.smithi064.stdout:5642 read 0x6f87e3f thru 0x6f8ef2b (0x70ed bytes) 2024-04-24T00:37:06.674 INFO:teuthology.orchestra.run.smithi064.stdout:5643 read 0x40f927c thru 0x40ffaa3 (0x6828 bytes) 2024-04-24T00:37:06.674 INFO:teuthology.orchestra.run.smithi064.stdout:5645 writesame 0xa077b24 thru 0xa07e954 data_size 0x1(0x6e31 bytes) 2024-04-24T00:37:06.686 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:37:06.692 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:06.692 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:06.709 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:06.724 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:06.725 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:06.739 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:06.739 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:06.740 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:06.740 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:06.754 INFO:teuthology.orchestra.run.smithi064.stdout:5646 write 0xad50d7c thru 0xad5885a (0x7adf bytes) 2024-04-24T00:37:06.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:06.759 INFO:teuthology.orchestra.run.smithi064.stdout:5647 write 0x5cd1c0f thru 0x5cd8848 (0x6c3a bytes) 2024-04-24T00:37:06.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:06.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:5648 write 0x16dd9f7 thru 0x16e82c9 (0xa8d3 bytes) 2024-04-24T00:37:06.775 INFO:teuthology.orchestra.run.smithi064.stdout:5649 write 0x319ef29 thru 0x31a657e (0x7656 bytes) 2024-04-24T00:37:06.781 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:06.784 INFO:teuthology.orchestra.run.smithi064.stdout:5650 read 0x6171802 thru 0x617654f (0x4d4e bytes) 2024-04-24T00:37:06.794 INFO:teuthology.orchestra.run.smithi064.stdout:5651 trunc from 0xad5885b to 0x3c6caac 2024-04-24T00:37:06.805 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:06.802+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:06.806 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:06.811 INFO:teuthology.orchestra.run.smithi064.stdout:5652 write 0x1950c4 thru 0x19de94 (0x8dd1 bytes) 2024-04-24T00:37:06.826 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:06.831 INFO:teuthology.orchestra.run.smithi064.stdout:5654 writesame 0xc7a1bf4 thru 0xc7a5682 data_size 0x13(0x3a8f bytes) 2024-04-24T00:37:06.831 INFO:teuthology.orchestra.run.smithi064.stdout:5655 punch from 0xb345e8b to 0xb3512e9, (0xb45e bytes) 2024-04-24T00:37:06.831 INFO:teuthology.orchestra.run.smithi064.stdout:5656 writesame 0x97e8555 thru 0x97ea464 data_size 0xe(0x1f10 bytes) 2024-04-24T00:37:06.835 INFO:teuthology.orchestra.run.smithi064.stdout:5657 writesame 0x8c9602 thru 0x8d2dab data_size 0x1b(0x97aa bytes) 2024-04-24T00:37:06.848 INFO:teuthology.orchestra.run.smithi064.stdout:5658 write 0xcbf625c thru 0xcbfbf70 (0x5d15 bytes) 2024-04-24T00:37:06.854 INFO:teuthology.orchestra.run.smithi064.stdout:5659 trunc from 0xcbfbf71 to 0xe91889d 2024-04-24T00:37:06.861 INFO:teuthology.orchestra.run.smithi064.stdout:5660 write 0x1384c0b thru 0x138f8aa (0xaca0 bytes) 2024-04-24T00:37:06.866 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:06.863+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:06.867 INFO:teuthology.orchestra.run.smithi064.stdout:5663 trunc from 0xe91889d to 0xeb2a73d 2024-04-24T00:37:06.873 INFO:teuthology.orchestra.run.smithi064.stdout:5664 read 0xae7d317 thru 0xae87bf4 (0xa8de bytes) 2024-04-24T00:37:06.873 INFO:teuthology.orchestra.run.smithi064.stdout:5665 write 0xc722939 thru 0xc7293b7 (0x6a7f bytes) 2024-04-24T00:37:06.879 INFO:teuthology.orchestra.run.smithi064.stdout:5667 read 0xd4a537b thru 0xd4a7a3b (0x26c1 bytes) 2024-04-24T00:37:06.880 INFO:teuthology.orchestra.run.smithi064.stdout:5668 writesame 0xd066bdc thru 0xd073302 data_size 0x1(0xc727 bytes) 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157092 3.7 3:e0648a57:::rbd_object_map.115ee709dad1:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.880592+0000", 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0033725389999999999, 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034236090000000002, 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:06.884 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157092 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.880592+0000", 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.880592+0000", 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.880594+0000", 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.201e-06 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.880600+0000", 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4389999999999997e-06 2024-04-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.880615+0000", 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.535e-05 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.880624+0000", 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9099999999999994e-06 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.880724+0000", 2024-04-24T00:37:06.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9728000000000005e-05 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.880881+0000", 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156642 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.881490+0000", 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00060935799999999995 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.883004+0000", 2024-04-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001513679 2024-04-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:06.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:06.968 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:06.963+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:07.007 INFO:teuthology.orchestra.run.smithi064.stdout:5669 write 0xe645a8f thru 0xe64a3b5 (0x4927 bytes) 2024-04-24T00:37:07.019 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.019 INFO:teuthology.orchestra.run.smithi064.stdout:5672 read 0x6a901d2 thru 0x6a9ebbd (0xe9ec bytes) 2024-04-24T00:37:07.019 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.019 INFO:teuthology.orchestra.run.smithi064.stdout:5674 write 0x6a1bc18 thru 0x6a2ae0c (0xf1f5 bytes) 2024-04-24T00:37:07.034 INFO:teuthology.orchestra.run.smithi064.stdout:5675 read 0x9cd7bd5 thru 0x9ce1be6 (0xa012 bytes) 2024-04-24T00:37:07.034 INFO:teuthology.orchestra.run.smithi064.stdout:5676 read 0xb02f95d thru 0xb0357bd (0x5e61 bytes) 2024-04-24T00:37:07.034 INFO:teuthology.orchestra.run.smithi064.stdout:5677 write 0x40da5de thru 0x40e2d13 (0x8736 bytes) 2024-04-24T00:37:07.045 INFO:teuthology.orchestra.run.smithi064.stdout:5678 writesame 0x7fe9131 thru 0x7ff57ea data_size 0x3(0xc6ba bytes) 2024-04-24T00:37:07.066 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:07.066 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:07.066 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:07.066 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:07.066 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:07.069 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:07.063+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:07.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi064.stdout:5679 read 0x295eae3 thru 0x296abfa (0xc118 bytes) 2024-04-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi064.stdout:5680 read 0xd8e2480 thru 0xd8e7b6a (0x56eb bytes) 2024-04-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi064.stdout:5682 write 0x7224db6 thru 0x722ab9e (0x5de9 bytes) 2024-04-24T00:37:07.136 INFO:teuthology.orchestra.run.smithi064.stdout:5683 punch from 0x3f4b5d5 to 0x3f52db7, (0x77e2 bytes) 2024-04-24T00:37:07.136 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.136 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.136 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.136 INFO:teuthology.orchestra.run.smithi064.stdout:5687 punch from 0xdfe9c9c to 0xdff0600, (0x6964 bytes) 2024-04-24T00:37:07.136 INFO:teuthology.orchestra.run.smithi064.stdout:5688 writesame 0x1788c83 thru 0x179638c data_size 0x1e(0xd70a bytes) 2024-04-24T00:37:07.170 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:07.166+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:07.176 INFO:teuthology.orchestra.run.smithi064.stdout:5689 trunc from 0xeb2a73d to 0x589673f 2024-04-24T00:37:07.200 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.200 INFO:teuthology.orchestra.run.smithi064.stdout:5694 punch from 0x237fa00 to 0x2382773, (0x2d73 bytes) 2024-04-24T00:37:07.208 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:07.208 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:07.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 159.49842228700001, 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.209 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:37:07.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 159.49832436200001, 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.214 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:37:07.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:37:07.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:37:07.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.220 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.13894483000001, 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:37:07.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.728200704, 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:37:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:37:07.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:37:07.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:37:07.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:37:07.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:37:07.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.233 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.70293209800001, 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:07.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.84972837799999, 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:37:07.238 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:37:07.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:37:07.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:37:07.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:37:07.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:37:07.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:37:07.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:37:07.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:37:07.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:37:07.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:37:07.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:37:07.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:37:07.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:07.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:37:07.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:37:07.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:37:07.249 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.817123264, 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:07.253 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.41254857, 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:07.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:37:07.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:37:07.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.260 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.35470834, 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:37:07.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:37:07.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:37:07.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 78.513145991000002, 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:37:07.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:37:07.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:37:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.534398439999997, 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.270 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:37:07.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:37:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:37:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.504756387, 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.278 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:37:07.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:37:07.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:37:07.281 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.358240975999998, 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:37:07.282 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:37:07.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:37:07.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:37:07.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.569468418, 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.546325005, 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:37:07.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.673016257, 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:37:07.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.620333979999998, 2024-04-24T00:37:07.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:37:07.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:37:07.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127274 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.539249121000001, 2024-04-24T00:37:07.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071400004000000003, 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127274 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:07.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667542+0000", 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667545+0000", 2024-04-24T00:37:07.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9059999999999998e-06 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667553+0000", 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4910000000000005e-06 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667560+0000", 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5339999999999996e-06 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738552+0000", 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070991992000000004 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738615+0000", 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3559000000000006e-05 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738941+0000", 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032590199999999998 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127275 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.539221406999999, 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071850050999999998, 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127275 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.314 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:07.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4000000000000002e-07 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667572+0000", 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9780000000000001e-06 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667581+0000", 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7080000000000003e-06 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667587+0000", 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738971+0000", 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071383263000000002 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739053+0000", 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2591999999999998e-05 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739419+0000", 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036566600000000001 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:128046 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.477792900000001, 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072176436999999996, 2024-04-24T00:37:07.318 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 128046 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728998+0000", 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8299999999999998e-07 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729000+0000", 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2479999999999999e-06 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729006+0000", 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9569999999999999e-06 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729010+0000", 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4730000000000002e-06 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800122+0000", 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071111555000000007 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800239+0000", 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117401 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.801174+0000", 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093442000000000004 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.322 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.323 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.323 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.323 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:07.323 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:07.324 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:07.324 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-24T00:37:07.324 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:37:07.324 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:37:07.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-04-24T00:37:07.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:07.327 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:07.318+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:07.327 INFO:teuthology.orchestra.run.smithi064.stdout:5695 write 0x971a41c thru 0x9722216 (0x7dfb bytes) 2024-04-24T00:37:07.327 INFO:teuthology.orchestra.run.smithi064.stdout:5696 write 0x6405646 thru 0x64057d4 (0x18f bytes) 2024-04-24T00:37:07.327 INFO:teuthology.orchestra.run.smithi064.stdout:5697 punch from 0x6244dfb to 0x624b873, (0x6a78 bytes) 2024-04-24T00:37:07.327 INFO:teuthology.orchestra.run.smithi064.stdout:5698 read 0x3ac0ecd thru 0x3ac37df (0x2913 bytes) 2024-04-24T00:37:07.327 INFO:teuthology.orchestra.run.smithi064.stdout:5700 trunc from 0x9722217 to 0x36b3864 2024-04-24T00:37:07.327 INFO:teuthology.orchestra.run.smithi064.stdout:5701 write 0xe538d3 thru 0xe624eb (0xec19 bytes) 2024-04-24T00:37:07.327 INFO:teuthology.orchestra.run.smithi064.stdout:5702 trunc from 0x36b3864 to 0x8da3649 2024-04-24T00:37:07.350 INFO:teuthology.orchestra.run.smithi064.stdout:5703 writesame 0xbecc11c thru 0xbed8ed7 data_size 0x26(0xcdbc bytes) 2024-04-24T00:37:07.364 INFO:teuthology.orchestra.run.smithi064.stdout:5704 write 0xdce77a2 thru 0xdceb371 (0x3bd0 bytes) 2024-04-24T00:37:07.375 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.375 INFO:teuthology.orchestra.run.smithi064.stdout:5706 write 0xc74ec7b thru 0xc751c5f (0x2fe5 bytes) 2024-04-24T00:37:07.384 INFO:teuthology.orchestra.run.smithi064.stdout:5707 punch from 0xae9ead4 to 0xaeaa5de, (0xbb0a bytes) 2024-04-24T00:37:07.384 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.384 INFO:teuthology.orchestra.run.smithi064.stdout:5709 read 0x5d01815 thru 0x5d10a9c (0xf288 bytes) 2024-04-24T00:37:07.384 INFO:teuthology.orchestra.run.smithi064.stdout:5710 read 0x735ca65 thru 0x735cd62 (0x2fe bytes) 2024-04-24T00:37:07.384 INFO:teuthology.orchestra.run.smithi064.stdout:5713 writesame 0x9386ccf thru 0x938a0c8 data_size 0x1(0x33fa bytes) 2024-04-24T00:37:07.405 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.405 INFO:teuthology.orchestra.run.smithi064.stdout:5715 punch from 0xb892ebe to 0xb8a1b01, (0xec43 bytes) 2024-04-24T00:37:07.405 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.417 INFO:teuthology.orchestra.run.smithi064.stdout:5717 trunc from 0xdceb372 to 0x623ce8c 2024-04-24T00:37:07.425 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:07.422+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:07.456 INFO:teuthology.orchestra.run.smithi064.stdout:5718 write 0x7c8b0c6 thru 0x7c9138d (0x62c8 bytes) 2024-04-24T00:37:07.469 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:07.469 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "" 2024-04-24T00:37:07.469 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:07.479 INFO:teuthology.orchestra.run.smithi064.stdout:5719 writesame 0x808ec10 thru 0x8091305 data_size 0x1(0x26f6 bytes) 2024-04-24T00:37:07.479 INFO:teuthology.orchestra.run.smithi064.stdout:5720 read 0x68d279c thru 0x68e1054 (0xe8b9 bytes) 2024-04-24T00:37:07.479 INFO:teuthology.orchestra.run.smithi064.stdout:5721 punch from 0x5882b70 to 0x588bd8f, (0x921f bytes) 2024-04-24T00:37:07.499 INFO:teuthology.orchestra.run.smithi064.stdout:5722 writesame 0xb877bd3 thru 0xb882583 data_size 0x1(0xa9b1 bytes) 2024-04-24T00:37:07.500 INFO:teuthology.orchestra.run.smithi064.stdout:5723 punch from 0x4e1296d to 0x4e1f0b8, (0xc74b bytes) 2024-04-24T00:37:07.500 INFO:teuthology.orchestra.run.smithi064.stdout:5724 write 0x79eb6bf thru 0x79f62d9 (0xac1b bytes) 2024-04-24T00:37:07.509 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157230 3.a 3:55bb54ef:::rbd_data.115ee709dad1.000000000000007b:head [set-alloc-hint object_size 1048576 write_size 1048576,write 439999~44059 in=44059b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:07.504760+0000", 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0048314509999999996, 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048553520000000003, 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.510 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157230 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.504760+0000", 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.504760+0000", 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.504764+0000", 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.504915+0000", 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015081599999999999 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.504948+0000", 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3856999999999998e-05 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.504972+0000", 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3445999999999999e-05 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.505099+0000", 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012733899999999999 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.505370+0000", 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027101499999999999 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.505603+0000", 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023248100000000001 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:37:07.514 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:07.515 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.515 INFO:teuthology.orchestra.run.smithi064.stdout:5726 write 0xa7b722f thru 0xa7c6153 (0xef25 bytes) 2024-04-24T00:37:07.525 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:07.522+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:07.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:07.528 INFO:teuthology.orchestra.run.smithi064.stdout:5727 write 0x7490972 thru 0x749dd0d (0xd39c bytes) 2024-04-24T00:37:07.541 INFO:teuthology.orchestra.run.smithi064.stdout:5728 read 0x36d7fcb thru 0x36dfa08 (0x7a3e bytes) 2024-04-24T00:37:07.541 INFO:teuthology.orchestra.run.smithi064.stdout:5730 read 0x54e67a3 thru 0x54e9473 (0x2cd1 bytes) 2024-04-24T00:37:07.541 INFO:teuthology.orchestra.run.smithi064.stdout:5732 read 0x28206eb thru 0x2825168 (0x4a7e bytes) 2024-04-24T00:37:07.568 INFO:teuthology.orchestra.run.smithi064.stdout:5734 trunc from 0xb882584 to 0x108a24e 2024-04-24T00:37:07.606 INFO:teuthology.orchestra.run.smithi064.stdout:5735 trunc from 0x108a24e to 0x1cb5600 2024-04-24T00:37:07.627 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:07.623+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi064.stdout:5736 writesame 0xee5ad37 thru 0xee5efc3 data_size 0x9(0x428d bytes) 2024-04-24T00:37:07.669 INFO:teuthology.orchestra.run.smithi064.stdout:5738 writesame 0x4e9502c thru 0x4ea1128 data_size 0x29(0xc0fd bytes) 2024-04-24T00:37:07.690 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:07.691 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:07.691 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:07.691 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:07.691 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:07.700 INFO:teuthology.orchestra.run.smithi064.stdout:5740 trunc from 0xee5efc4 to 0x8182b16 2024-04-24T00:37:07.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:07.714 INFO:teuthology.orchestra.run.smithi064.stdout:5741 read 0x535889b thru 0x5364ece (0xc634 bytes) 2024-04-24T00:37:07.715 INFO:teuthology.orchestra.run.smithi064.stdout:5743 write 0x2fbc4d3 thru 0x2fbdc7e (0x17ac bytes) 2024-04-24T00:37:07.721 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.721 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:07.727 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:07.723+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:07.728 INFO:teuthology.orchestra.run.smithi064.stdout:5746 trunc from 0x8182b16 to 0xaffc5aa 2024-04-24T00:37:07.742 INFO:teuthology.orchestra.run.smithi064.stdout:5749 clone 38 order 19 su 4096 sc 8 2024-04-24T00:37:07.829 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:07.823+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:07.831 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:07.832 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:07.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:07.832 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:07.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.832 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:37:07.833 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:07.833 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.29730976600001, 2024-04-24T00:37:07.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:37:07.833 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.833 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.833 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.834 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:37:07.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.834 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:07.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:07.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:37:07.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:37:07.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:37:07.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:37:07.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:37:07.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:37:07.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:37:07.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:37:07.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:37:07.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:07.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:37:07.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:37:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:37:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:37:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:37:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:37:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:37:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:37:07.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:37:07.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:37:07.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.844 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.26327271299999, 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:07.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:37:07.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:37:07.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:37:07.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:37:07.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 137.148158717, 2024-04-24T00:37:07.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:07.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:37:07.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 137.120708756, 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:37:07.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.200636923999994, 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:07.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:37:07.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:37:07.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:37:07.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:37:07.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:37:07.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:37:07.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:37:07.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:37:07.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:37:07.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:37:07.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:37:07.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:07.867 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.194365665999996, 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:37:07.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:37:07.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:37:07.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:37:07.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:37:07.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:37:07.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:37:07.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:37:07.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:37:07.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:37:07.875 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.875 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.875 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.875 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:37:07.876 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:07.876 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.182911079999997, 2024-04-24T00:37:07.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:37:07.876 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.876 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.876 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.876 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:07.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:07.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:37:07.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:37:07.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:37:07.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:37:07.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:37:07.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:37:07.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:37:07.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:37:07.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:37:07.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:37:07.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:37:07.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:37:07.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:37:07.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:37:07.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:37:07.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:37:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:37:07.886 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:07.886 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.128458535999997, 2024-04-24T00:37:07.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:37:07.886 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.886 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.886 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.887 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:07.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:07.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:37:07.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:37:07.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:37:07.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:37:07.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:37:07.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:37:07.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:37:07.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:37:07.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:37:07.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:37:07.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:37:07.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105171 3.1e 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head [zero 0~262144,write 262144~131072 [fadvise_sequential+fadvise_nocache],zero 393216~131072,write 524288~917504 [fadvise_sequential+fadvise_nocache],zero 1441792~393216,write 1835008~131072 [fadvise_sequential+fadvise_nocache],write 2097152~917504 [fadvise_sequential+fadvise_nocache],zero 3014656~393216,write 3407872~131072 [fadvise_sequential+fadvise_nocache],zero 3538944~131072,write 3670016~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.138708043999998, 2024-04-24T00:37:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061400402, 2024-04-24T00:37:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105171 2024-04-24T00:37:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690858+0000", 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5059999999999998e-06 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697752+0000", 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006893766 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697795+0000", 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2589000000000002e-05 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697810+0000", 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5620999999999998e-05 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697936+0000", 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012529199999999999 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698150+0000", 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021430200000000001 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698445+0000", 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029468099999999998 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736712+0000", 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038267781000000001 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752153+0000", 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015441043999999999 2024-04-24T00:37:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752204+0000", 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752256+0000", 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2286e-05 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127048 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 378880~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.179515397999999, 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768986, 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127048 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.92e-07 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650053+0000", 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.698e-06 2024-04-24T00:37:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650065+0000", 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1454000000000001e-05 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650073+0000", 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8920000000000006e-06 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705378+0000", 2024-04-24T00:37:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055304820999999997 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705456+0000", 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8009999999999993e-05 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705818+0000", 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036131899999999997 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127049 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 382976~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.179478226000001, 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056235535000000003, 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127049 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650090+0000", 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650101+0000", 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0822000000000001e-05 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650110+0000", 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705851+0000", 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055741377000000002 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705916+0000", 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5160000000000006e-05 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706321+0000", 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040486500000000002 2024-04-24T00:37:07.912 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127050 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 387072~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.179442765000001, 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056659432000000003, 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.913 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127050 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650122+0000", 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650125+0000", 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.366e-06 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650136+0000", 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0624e-05 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650144+0000", 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4360000000000002e-06 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706354+0000", 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056210006999999999 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706437+0000", 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2515000000000005e-05 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706781+0000", 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034373599999999998 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127051 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.179408812999998, 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057055097999999999, 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.917 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127051 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:07.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650156+0000", 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1399999999999996e-07 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650159+0000", 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3100000000000001e-06 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650170+0000", 2024-04-24T00:37:07.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.058e-05 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650178+0000", 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5660000000000007e-06 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706827+0000", 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056648506000000001 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706903+0000", 2024-04-24T00:37:07.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6493000000000001e-05 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.707210+0000", 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306929 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127335 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.165093197000001, 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055806472000000003, 2024-04-24T00:37:07.921 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127335 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5200000000000002e-07 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664474+0000", 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6400000000000001e-06 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664481+0000", 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.002e-06 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664493+0000", 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2311e-05 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719879+0000", 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055386187000000003 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719956+0000", 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6196e-05 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720277+0000", 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032168399999999998 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127336 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.925 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.165060685, 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056212985, 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127336 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664504+0000", 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5400000000000001e-07 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664513+0000", 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.132e-06 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664520+0000", 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3849999999999996e-06 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664525+0000", 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0180000000000001e-06 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720371+0000", 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055845976999999998 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720430+0000", 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8964e-05 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720716+0000", 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028595500000000001 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.929 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127337 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.165029312000001, 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056616423999999999, 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127337 2024-04-24T00:37:07.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:07.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2099999999999997e-07 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664538+0000", 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0410000000000002e-06 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664546+0000", 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3710000000000002e-06 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664551+0000", 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0889999999999999e-06 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720759+0000", 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056208658000000002 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720811+0000", 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1564999999999998e-05 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721151+0000", 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034017900000000002 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127338 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.165005172000001, 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057296623999999997, 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.934 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127338 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.58e-07 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664562+0000", 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.345e-06 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664568+0000", 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-04-24T00:37:07.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664573+0000", 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721197+0000", 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056624102000000003 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721296+0000", 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9001999999999998e-05 2024-04-24T00:37:07.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721855+0000", 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055934300000000001 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127339 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 23.164984625999999, 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057818889999999998, 2024-04-24T00:37:07.938 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127339 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664580+0000", 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5299999999999999e-07 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664582+0000", 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1399999999999998e-06 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664588+0000", 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1959999999999996e-06 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664593+0000", 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.656e-06 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721894+0000", 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057301501999999997 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721991+0000", 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6330999999999996e-05 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.722398+0000", 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040761200000000001 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.942 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.453609901, 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 17.432718354999999, 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:07.948 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:37:07.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:07.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:07.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:37:07.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:37:07.952 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.952 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.952 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.952 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:07.952 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:07.952 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:07.954 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:07.954 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:07.950+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:08.054 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:08.051+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:08.126 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:08.126 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:08.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:08.186 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:08.182+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:08.256 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:08.251+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:08.261 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:08.261 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:08.261 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:08.262 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:08.262 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:08.276 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:08.357 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:08.355+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.50701200099999, 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.437 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:37:08.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.575046313, 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.57379756500001, 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.449 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:37:08.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 165.416173461, 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:37:08.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:37:08.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 165.40267891799999, 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.464 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 161.140897803, 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.88731165999999, 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.470 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:37:08.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:37:08.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.85963779599999, 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.476 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 147.36561497599999, 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.481 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:37:08.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:37:08.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:37:08.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 147.356606142, 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:37:08.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.85490719400001, 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.491 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:37:08.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:37:08.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.854844848, 2024-04-24T00:37:08.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:37:08.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.854335169, 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.85422410699999, 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.504 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:08.508 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.854158841, 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:37:08.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.854098775, 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.513 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:37:08.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:37:08.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.85404208200001, 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.517 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.521 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.85397274899999, 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:37:08.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:37:08.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.85388213, 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.526 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:37:08.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:37:08.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.853838717, 2024-04-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:37:08.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.534 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:08.535 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:08.535 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:08.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:08.537 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:08.530+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:08.638 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:08.635+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:08.673 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:08.673 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:08.674 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:08.674 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:08.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:08.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:37:08.773 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:08.771+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:08.839 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:08.835+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:08.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:08.853 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:08.854 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:08.854 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:08.854 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:08.854 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:08.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:08.940 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:08.934+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:08.988 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:08.989 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:09.005 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:09.006 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:09.022 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:09.023 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:09.035 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:09.036 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:09.041 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:09.041 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:09.041 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:09.039+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:09.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:09.062 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:09.062 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:09.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:09.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:09.104 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:09.104 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:09.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:09.126 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:09.142 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:09.138+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:09.179 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:09.179 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:09.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:09.243 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:09.242+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:09.344 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:09.339+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:09.354 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:09.354 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:09.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:09.445 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:09.442+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:09.524 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:09.525 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:09.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:09.546 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:09.542+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:09.581 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:37:09.653 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:09.651+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:09.667 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:09.667 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:09.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:09.773 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:09.771+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:09.848 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:09.843+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:09.880 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:09.880 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:09.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:09.985 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:09.983+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:10.050 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:10.046+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:10.070 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:10.070 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:10.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:10.152 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:10.146+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:10.248 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:10.248 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:10.252 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:10.250+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:10.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:10.353 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:10.351+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:10.387 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:10.387 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:10.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:10.454 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:10.454+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:10.555 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:10.551+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:10.573 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:10.573 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:10.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:10.657 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:10.654+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:10.727 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:10.727 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:10.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:10.758 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:10.755+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:10.860 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:10.855+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:10.910 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:10.910 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:10.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:10.937 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #36, image image_client.0-clone36 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent37 2024-04-24T00:37:10.973 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:10.970+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:11.051 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:11.051 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:11.061 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:11.055+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:11.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:11.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:11.194 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:11.191+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:11.226 INFO:teuthology.orchestra.run.smithi064.stdout:5750 write 0x58db252 thru 0x58dea56 (0x3805 bytes) 2024-04-24T00:37:11.228 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:11.228 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:11.245 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:11.249 INFO:teuthology.orchestra.run.smithi064.stdout:5751 trunc from 0xaffc5aa to 0xcf1c57f 2024-04-24T00:37:11.259 INFO:teuthology.orchestra.run.smithi064.stdout:5753 write 0xbdfe43d thru 0xbe0141b (0x2fdf bytes) 2024-04-24T00:37:11.263 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:11.259+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:11.287 INFO:teuthology.orchestra.run.smithi064.stdout:5755 trunc from 0xcf1c57f to 0x12d0a57 2024-04-24T00:37:11.312 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:11.312 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:11.330 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:11.346 INFO:teuthology.orchestra.run.smithi064.stdout:5756 write 0xaf65a50 thru 0xaf6ac0b (0x51bc bytes) 2024-04-24T00:37:11.348 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:11.348 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:11.357 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.357 INFO:teuthology.orchestra.run.smithi064.stdout:5758 write 0x30393ac thru 0x30399cb (0x620 bytes) 2024-04-24T00:37:11.361 INFO:teuthology.orchestra.run.smithi064.stdout:5760 write 0x12848fa thru 0x129022a (0xb931 bytes) 2024-04-24T00:37:11.365 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:11.358+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:11.368 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:11.368 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:11.369 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.369 INFO:teuthology.orchestra.run.smithi064.stdout:5762 write 0x225b4a thru 0x22d9ce (0x7e85 bytes) 2024-04-24T00:37:11.377 INFO:teuthology.orchestra.run.smithi064.stdout:5763 writesame 0x89bfbe5 thru 0x89cdfe8 data_size 0x2(0xe404 bytes) 2024-04-24T00:37:11.377 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:11.377 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:11.392 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:11.392 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:11.394 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:11.399 INFO:teuthology.orchestra.run.smithi064.stdout:5765 write 0x606b097 thru 0x607701e (0xbf88 bytes) 2024-04-24T00:37:11.413 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:11.414 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:11.415 INFO:teuthology.orchestra.run.smithi064.stdout:5766 read 0x2044758 thru 0x2053971 (0xf21a bytes) 2024-04-24T00:37:11.415 INFO:teuthology.orchestra.run.smithi064.stdout:5767 write 0xa68efdf thru 0xa69aeb9 (0xbedb bytes) 2024-04-24T00:37:11.432 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.432 INFO:teuthology.orchestra.run.smithi064.stdout:5769 trunc from 0xaf6ac0c to 0xb24edec 2024-04-24T00:37:11.434 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:11.434 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:11.439 INFO:teuthology.orchestra.run.smithi064.stdout:5770 read 0x4ff51a1 thru 0x5004110 (0xef70 bytes) 2024-04-24T00:37:11.439 INFO:teuthology.orchestra.run.smithi064.stdout:5771 write 0x69bda1c thru 0x69bfa09 (0x1fee bytes) 2024-04-24T00:37:11.447 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.447 INFO:teuthology.orchestra.run.smithi064.stdout:5773 writesame 0x620f783 thru 0x6210bad data_size 0x1(0x142b bytes) 2024-04-24T00:37:11.451 INFO:teuthology.orchestra.run.smithi064.stdout:5774 read 0x5f9f450 thru 0x5faec89 (0xf83a bytes) 2024-04-24T00:37:11.452 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.452 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.452 INFO:teuthology.orchestra.run.smithi064.stdout:5777 read 0x1154e01 thru 0x115c519 (0x7719 bytes) 2024-04-24T00:37:11.452 INFO:teuthology.orchestra.run.smithi064.stdout:5778 read 0x16810bf thru 0x168144c (0x38e bytes) 2024-04-24T00:37:11.453 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.453 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.453 INFO:teuthology.orchestra.run.smithi064.stdout:5783 writesame 0x3cca46d thru 0x3cd74b1 data_size 0x25(0xd045 bytes) 2024-04-24T00:37:11.453 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:11.467 INFO:teuthology.orchestra.run.smithi064.stdout:5784 read 0x308c943 thru 0x3091b70 (0x522e bytes) 2024-04-24T00:37:11.468 INFO:teuthology.orchestra.run.smithi064.stdout:5785 writesame 0x2a59b48 thru 0x2a668a4 data_size 0x1(0xcd5d bytes) 2024-04-24T00:37:11.484 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:11.478+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:11.503 INFO:teuthology.orchestra.run.smithi064.stdout:5786 read 0x32b2e94 thru 0x32b91bf (0x632c bytes) 2024-04-24T00:37:11.503 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.503 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:11.517 INFO:teuthology.orchestra.run.smithi064.stdout:5791 write 0xe96f115 thru 0xe96fe2b (0xd17 bytes) 2024-04-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi064.stdout:5792 punch from 0x8c22c3b to 0x8c29bb3, (0x6f78 bytes) 2024-04-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi064.stdout:5793 punch from 0x44a492c to 0x44a9b88, (0x525c bytes) 2024-04-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi064.stdout:5797 read 0xbe74787 thru 0xbe81a16 (0xd290 bytes) 2024-04-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi064.stdout:5798 read 0x55e2b1c thru 0x55eb1dc (0x86c1 bytes) 2024-04-24T00:37:11.522 INFO:teuthology.orchestra.run.smithi064.stdout:5799 writesame 0x7244943 thru 0x724712c data_size 0x1a(0x27ea bytes) 2024-04-24T00:37:11.529 INFO:teuthology.orchestra.run.smithi064.stdout:5800 read 0x6cda9a9 thru 0x6ce2c43 (0x829b bytes) 2024-04-24T00:37:11.529 INFO:teuthology.orchestra.run.smithi064.stdout:5801 writesame 0x4f58c74 thru 0x4f5e737 data_size 0x4(0x5ac4 bytes) 2024-04-24T00:37:11.541 INFO:teuthology.orchestra.run.smithi064.stdout:5802 read 0x5f18218 thru 0x5f24985 (0xc76e bytes) 2024-04-24T00:37:11.542 INFO:teuthology.orchestra.run.smithi064.stdout:5803 read 0x48621e thru 0x48b816 (0x55f9 bytes) 2024-04-24T00:37:11.542 INFO:teuthology.orchestra.run.smithi064.stdout:5804 clone 39 order 21 su 131072 sc 2 2024-04-24T00:37:11.566 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:11.563+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:11.568 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:11.568 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:11.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:11.568 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:11.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 163.85793438799999, 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:37:11.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:37:11.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.574 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 163.85783646300001, 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:11.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:37:11.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:37:11.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:11.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:37:11.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:37:11.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:37:11.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:37:11.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:37:11.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:37:11.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:37:11.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.583 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.49845693099999, 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.584 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:37:11.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.087712805, 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:37:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:37:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:37:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:37:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.062444199, 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:37:11.597 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:37:11.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:37:11.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.601 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.209240479, 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:37:11.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:11.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:37:11.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.176635365, 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:37:11.608 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:37:11.612 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.77206067100001, 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.613 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:11.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:37:11.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:11.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.71422044099999, 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.619 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:37:11.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.623 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.624 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.624 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89924 3.f e125/123 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head v 125'254, mlcod=124'252)", 2024-04-24T00:37:11.624 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:11.624 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 82.872658091999995, 2024-04-24T00:37:11.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071466294, 2024-04-24T00:37:11.625 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.625 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.625 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.625 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:11.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693644+0000", 2024-04-24T00:37:11.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.693645+0000", 2024-04-24T00:37:11.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2199999999999999e-07 2024-04-24T00:37:11.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699172+0000", 2024-04-24T00:37:11.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055270700000000002 2024-04-24T00:37:11.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699203+0000", 2024-04-24T00:37:11.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1696999999999998e-05 2024-04-24T00:37:11.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699219+0000", 2024-04-24T00:37:11.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5937e-05 2024-04-24T00:37:11.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699379+0000", 2024-04-24T00:37:11.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015998 2024-04-24T00:37:11.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.699440+0000", 2024-04-24T00:37:11.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0273999999999999e-05 2024-04-24T00:37:11.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765060+0000", 2024-04-24T00:37:11.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065620600000000001 2024-04-24T00:37:11.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765110+0000", 2024-04-24T00:37:11.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0314e-05 2024-04-24T00:37:11.633 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.633 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.633 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.634 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:11.634 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:11.634 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.893910540999997, 2024-04-24T00:37:11.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:37:11.634 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.635 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.635 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.635 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.635 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.635 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.635 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:37:11.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.636 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:11.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:11.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:37:11.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:37:11.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:37:11.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:37:11.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:37:11.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:37:11.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:37:11.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:37:11.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:37:11.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:37:11.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:37:11.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:37:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:37:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:11.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:37:11.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:37:11.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.648 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.648 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:37:11.649 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:11.649 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.864268487999993, 2024-04-24T00:37:11.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:37:11.649 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.649 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.649 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.649 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:11.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:37:11.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:37:11.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.717753076999998, 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.654 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:37:11.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:37:11.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:37:11.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:37:11.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:37:11.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:37:11.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:37:11.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.661 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.662 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:11.662 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:11.662 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.928980519, 2024-04-24T00:37:11.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:37:11.663 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.663 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.663 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.664 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:37:11.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.664 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:11.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:11.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:37:11.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:37:11.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:37:11.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:37:11.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:37:11.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:37:11.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:37:11.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:37:11.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:37:11.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:37:11.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:37:11.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:37:11.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:11.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:37:11.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:37:11.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:11.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:37:11.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:11.673 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.905837106, 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:37:11.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:37:11.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 51.032528358, 2024-04-24T00:37:11.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:37:11.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:37:11.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:37:11.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.979846080999998, 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:37:11.683 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:37:11.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127274 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:11.687 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.898761222000001, 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071400004000000003, 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127274 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:11.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667541+0000", 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667542+0000", 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1999999999999999e-07 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667545+0000", 2024-04-24T00:37:11.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9059999999999998e-06 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667553+0000", 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4910000000000005e-06 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667560+0000", 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5339999999999996e-06 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738552+0000", 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070991992000000004 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738615+0000", 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3559000000000006e-05 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738941+0000", 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032590199999999998 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127275 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.898733507999999, 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071850050999999998, 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127275 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.692 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667569+0000", 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4000000000000002e-07 2024-04-24T00:37:11.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667572+0000", 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9780000000000001e-06 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667581+0000", 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7080000000000003e-06 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.667587+0000", 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.738971+0000", 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071383263000000002 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739053+0000", 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2591999999999998e-05 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.739419+0000", 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036566600000000001 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:128046 3.c 3:35d4d0ad:::rbd_data.115ee44a9909.0000000000000004:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.837305001000001, 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072176436999999996, 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:11.696 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 128046 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728997+0000", 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.728998+0000", 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8299999999999998e-07 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729000+0000", 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2479999999999999e-06 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729006+0000", 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9569999999999999e-06 2024-04-24T00:37:11.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.729010+0000", 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4730000000000002e-06 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800122+0000", 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071111555000000007 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.800239+0000", 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117401 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:11.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.801174+0000", 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093442000000000004 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:11.700 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:11.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:11.702 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:11.699+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:11.804 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:11.799+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:11.834 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:11.835 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:11.835 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:11.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:11.905 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:11.902+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:11.969 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:11.969 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:11.969 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:11.969 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:11.969 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:11.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:12.025 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:12.022+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:12.107 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:12.102+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:12.151 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:37:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:12.151 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 173.61521119, 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:37:12.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:37:12.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 173.581174137, 2024-04-24T00:37:12.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:12.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:37:12.162 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.46606014100001, 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.163 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:37:12.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:12.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:37:12.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.43861018000001, 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:37:12.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:37:12.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.174 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.518538348000007, 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:37:12.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:37:12.179 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.512267089999995, 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.180 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:37:12.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87729 3.3 e115/59 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head v 115'167, mlcod=108'165)", 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.500812503999995, 2024-04-24T00:37:12.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061305630999999999, 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646653+0000", 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646655+0000", 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.68e-06 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652166+0000", 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055110530000000001 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652186+0000", 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0324000000000001e-05 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652198+0000", 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2062000000000001e-05 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652281+0000", 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2562e-05 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.652306+0000", 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5517e-05 2024-04-24T00:37:12.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707893+0000", 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055587193 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.707959+0000", 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5240000000000006e-05 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87732 3.11 e115/106 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head v 115'359, mlcod=113'356)", 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.446359959999995, 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060589796000000001, 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701105+0000", 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.701107+0000", 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9240000000000001e-06 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705241+0000", 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041336990000000002 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705267+0000", 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5882000000000001e-05 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.705280+0000", 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.327e-05 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728900+0000", 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023620072999999998 2024-04-24T00:37:12.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.728925+0000", 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4848999999999999e-05 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761653+0000", 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032727807999999997 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761695+0000", 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2290999999999999e-05 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:105171 3.1e 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head [zero 0~262144,write 262144~131072 [fadvise_sequential+fadvise_nocache],zero 393216~131072,write 524288~917504 [fadvise_sequential+fadvise_nocache],zero 1441792~393216,write 1835008~131072 [fadvise_sequential+fadvise_nocache],write 2097152~917504 [fadvise_sequential+fadvise_nocache],zero 3014656~393216,write 3407872~131072 [fadvise_sequential+fadvise_nocache],zero 3538944~131072,write 3670016~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.456609468000003, 2024-04-24T00:37:12.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061400402, 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 105171 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:12.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690856+0000", 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.690858+0000", 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5059999999999998e-06 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697752+0000", 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006893766 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697795+0000", 2024-04-24T00:37:12.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2589000000000002e-05 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697810+0000", 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5620999999999998e-05 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.697936+0000", 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012529199999999999 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698150+0000", 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021430200000000001 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.698445+0000", 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029468099999999998 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.736712+0000", 2024-04-24T00:37:12.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038267781000000001 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752153+0000", 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015441043999999999 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752204+0000", 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0534000000000001e-05 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:15.752256+0000", 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2286e-05 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127048 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 378880~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.497416822000002, 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055768986, 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.205 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127048 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:12.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650049+0000", 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.92e-07 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650053+0000", 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.698e-06 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650065+0000", 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1454000000000001e-05 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650073+0000", 2024-04-24T00:37:12.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8920000000000006e-06 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705378+0000", 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055304820999999997 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705456+0000", 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8009999999999993e-05 2024-04-24T00:37:12.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705818+0000", 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00036131899999999997 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127049 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 382976~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.210 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.497379649999999, 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056235535000000003, 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127049 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650086+0000", 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6199999999999997e-07 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650090+0000", 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8360000000000002e-06 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650101+0000", 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0822000000000001e-05 2024-04-24T00:37:12.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650110+0000", 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7129999999999994e-06 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705851+0000", 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055741377000000002 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.705916+0000", 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5160000000000006e-05 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706321+0000", 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040486500000000002 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127050 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 387072~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.497344189, 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056659432000000003, 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.216 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127050 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650121+0000", 2024-04-24T00:37:12.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650122+0000", 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650125+0000", 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.366e-06 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650136+0000", 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0624e-05 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650144+0000", 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4360000000000002e-06 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706354+0000", 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056210006999999999 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706437+0000", 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2515000000000005e-05 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706781+0000", 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034373599999999998 2024-04-24T00:37:12.220 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127051 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 391168~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.497310237000001, 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057055097999999999, 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127051 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650155+0000", 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650156+0000", 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1399999999999996e-07 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650159+0000", 2024-04-24T00:37:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3100000000000001e-06 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650170+0000", 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.058e-05 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.650178+0000", 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5660000000000007e-06 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706827+0000", 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056648506000000001 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.706903+0000", 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6493000000000001e-05 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.707210+0000", 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000306929 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127335 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.482994621, 2024-04-24T00:37:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055806472000000003, 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127335 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664471+0000", 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5200000000000002e-07 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664474+0000", 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6400000000000001e-06 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664481+0000", 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.002e-06 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664493+0000", 2024-04-24T00:37:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2311e-05 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719879+0000", 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055386187000000003 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.719956+0000", 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6196e-05 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720277+0000", 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032168399999999998 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127336 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.482962108999999, 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056212985, 2024-04-24T00:37:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127336 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664503+0000", 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664504+0000", 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5400000000000001e-07 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664513+0000", 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.132e-06 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664520+0000", 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3849999999999996e-06 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664525+0000", 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0180000000000001e-06 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720371+0000", 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055845976999999998 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720430+0000", 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8964e-05 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720716+0000", 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028595500000000001 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127337 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.236 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.482930736, 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056616423999999999, 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127337 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664535+0000", 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2099999999999997e-07 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664538+0000", 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0410000000000002e-06 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664546+0000", 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3710000000000002e-06 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664551+0000", 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0889999999999999e-06 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720759+0000", 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056208658000000002 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.720811+0000", 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1564999999999998e-05 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721151+0000", 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034017900000000002 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127338 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.482906595999999, 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057296623999999997, 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127338 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664559+0000", 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.58e-07 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664562+0000", 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.345e-06 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664568+0000", 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5189999999999998e-06 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664573+0000", 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8550000000000001e-06 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721197+0000", 2024-04-24T00:37:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.056624102000000003 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721296+0000", 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9001999999999998e-05 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721855+0000", 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055934300000000001 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:127339 3.10 3:0bc2dd10:::rbd_data.115ee44a9909.0000000000000006:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 27.482886050000001, 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057818889999999998, 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 127339 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664579+0000", 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664580+0000", 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5299999999999999e-07 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664582+0000", 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1399999999999998e-06 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664588+0000", 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1959999999999996e-06 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.664593+0000", 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.656e-06 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721894+0000", 2024-04-24T00:37:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057301501999999997 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.721991+0000", 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6330999999999996e-05 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:44.722398+0000", 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040761200000000001 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.771511324999999, 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:37:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 21.750619779000001, 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:37:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.263 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:12.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:12.265 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:12.259+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:12.366 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:12.363+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:12.435 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:12.435 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:12.435 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:12.436 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:12.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:12.489 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:12.489 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:12.489 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-24T00:37:12.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-24T00:37:12.493 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:12.490+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:12.567 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:12.563+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:12.569 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:12.569 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:12.569 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:12.569 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:12.569 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:12.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:12.669 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:12.666+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:12.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-24T00:37:12.721 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:37:12.762 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:12.762 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:12.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:12.762 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:12.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.762 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 211.833618734, 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.768 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.90165304600001, 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.769 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:37:12.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:37:12.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.90040429800001, 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.775 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:37:12.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:12.780 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.74278019400001, 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:37:12.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:37:12.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.729285651, 2024-04-24T00:37:12.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:12.790 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 165.46750453600001, 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:12.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:37:12.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:37:12.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.213918393, 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.796 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:37:12.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:37:12.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:37:12.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.18624452899999, 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.802 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:12.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:37:12.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.69222170899999, 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.807 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:37:12.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.68321287500001, 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.813 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:37:12.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:12.817 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.181513927, 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.181451581, 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.822 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:12.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:37:12.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:12.827 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.180941902, 2024-04-24T00:37:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:37:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:37:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.18083084, 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:37:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:37:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.18076557399999, 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:37:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.18070550799999, 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.18064881500001, 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:37:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.18057948200001, 2024-04-24T00:37:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:37:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.18048886299999, 2024-04-24T00:37:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:37:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:37:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:37:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.18044545000001, 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:37:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:37:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:37:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:37:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:12.870 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:12.871 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:12.872 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:12.866+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:12.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-24T00:37:12.972 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:12.970+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:13.040 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:13.041 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:13.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:13.073 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:13.070+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:13.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-24T00:37:13.179 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:13.179 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:13.179 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:13.175+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:13.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:13.285 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:13.283+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:13.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110941 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 32768~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476866663999999, 2024-04-24T00:37:13.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99518982899999997, 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110941 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882159+0000", 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882161+0000", 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882169+0000", 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8480000000000001e-06 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882173+0000", 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6110000000000004e-06 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886136+0000", 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039630810000000002 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886147+0000", 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0484e-05 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876102+0000", 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98995489400000003 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876434+0000", 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033270799999999997 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.877348+0000", 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091357700000000005 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110942 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 229376~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476844917999998, 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99714419399999998, 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110942 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5199999999999998e-07 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882182+0000", 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8819999999999999e-06 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882190+0000", 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3629999999999996e-06 2024-04-24T00:37:13.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882194+0000", 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5000000000000001e-06 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886176+0000", 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039815329999999998 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886184+0000", 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-04-24T00:37:13.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878386+0000", 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99220202999999996 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878535+0000", 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001494 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879324+0000", 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078909599999999998 2024-04-24T00:37:13.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110958 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 65536~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476748606000001, 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.998192368, 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.373 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110958 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882277+0000", 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882278+0000", 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6419999999999999e-06 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882284+0000", 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3890000000000004e-06 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882288+0000", 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.138e-06 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886955+0000", 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046673289999999996 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886964+0000", 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8310000000000007e-06 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879623+0000", 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99265892499999997 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879818+0000", 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019486800000000001 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880469+0000", 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065093099999999999 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110959 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 262144~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476688707999998, 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99945926699999998, 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110959 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882337+0000", 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4499999999999997e-07 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882340+0000", 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0259999999999999e-06 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882358+0000", 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8689999999999999e-05 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882365+0000", 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3879999999999997e-06 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886989+0000", 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046244930000000004 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886997+0000", 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880603+0000", 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99360585199999996 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880779+0000", 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175792 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881795+0000", 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001016922 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110974 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 98304~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476541263000001, 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.000617986, 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110974 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2100000000000002e-07 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882486+0000", 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.215e-06 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882494+0000", 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9410000000000007e-06 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882500+0000", 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2389999999999999e-06 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888427+0000", 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059265029999999996 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888436+0000", 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2019999999999993e-06 2024-04-24T00:37:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881936+0000", 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99350027399999996 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.882146+0000", 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020958500000000001 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883102+0000", 2024-04-24T00:37:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095560600000000001 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110975 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 294912~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476517780999998, 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002014129, 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110975 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882508+0000", 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882510+0000", 2024-04-24T00:37:13.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.148e-06 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882515+0000", 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7289999999999997e-06 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882520+0000", 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3290000000000004e-06 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888463+0000", 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059433669999999997 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888470+0000", 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.187e-06 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883245+0000", 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99477520799999997 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883497+0000", 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025132699999999997 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884521+0000", 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001024435 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.394 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110990 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 131072~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476366642999999, 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0028161980000001, 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.395 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110990 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882659+0000", 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4299999999999998e-07 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882661+0000", 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-06 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882667+0000", 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.879e-06 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882671+0000", 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5349999999999998e-06 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889926+0000", 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072543970000000001 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889935+0000", 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7070000000000004e-06 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884669+0000", 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99473391700000002 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884841+0000", 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017212600000000001 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885474+0000", 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063308099999999996 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110991 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 327680~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476347590000003, 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0041004259999999, 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110991 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882678+0000", 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882684+0000", 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3219999999999998e-06 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882691+0000", 2024-04-24T00:37:13.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882705+0000", 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3813e-05 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889972+0000", 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072669309999999999 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889981+0000", 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3059999999999998e-06 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885568+0000", 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99558754299999996 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885798+0000", 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023032799999999999 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886778+0000", 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097934600000000008 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111006 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 163840~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476211798000001, 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0048379629999999, 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111006 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882815+0000", 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7549999999999999e-06 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882822+0000", 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0539999999999997e-06 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882827+0000", 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3619999999999999e-06 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890715+0000", 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078878809999999994 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890723+0000", 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6289999999999999e-06 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886854+0000", 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99613063000000002 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887032+0000", 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017786200000000001 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887651+0000", 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061942900000000001 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111007 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 360448~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.476192369000003, 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061255339999999, 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111007 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0400000000000002e-07 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882835+0000", 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7129999999999999e-06 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882842+0000", 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1280000000000002e-06 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882846+0000", 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5600000000000004e-06 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890749+0000", 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079023110000000004 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890758+0000", 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2070000000000001e-06 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887756+0000", 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99699837099999999 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888019+0000", 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263014 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888958+0000", 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093882600000000003 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111032 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.446258612999998, 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97659819800000003, 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.415 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111032 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912767+0000", 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912769+0000", 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6759999999999999e-06 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912777+0000", 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7659999999999998e-06 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912782+0000", 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0590000000000002e-06 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913091+0000", 2024-04-24T00:37:13.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030847099999999999 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913101+0000", 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889077+0000", 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.975975437 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889228+0000", 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015131699999999999 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889364+0000", 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013657900000000001 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111065 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.435463018, 2024-04-24T00:37:13.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96634889000000002, 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111065 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923563+0000", 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7100000000000001e-07 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923566+0000", 2024-04-24T00:37:13.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.658e-06 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923577+0000", 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268999999999999e-05 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923588+0000", 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1299e-05 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923668+0000", 2024-04-24T00:37:13.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0012000000000006e-05 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923678+0000", 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9019999999999994e-06 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889478+0000", 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96580058899999999 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889697+0000", 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218285 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889911+0000", 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000214105 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111116 3.1f 3:f85dfb34:::rbd_data.115e397a9cd4.000000000000000c:head [write 0~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.424261575999999, 2024-04-24T00:37:13.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96849046100000002, 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111116 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934766+0000", 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3580000000000001e-06 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935127+0000", 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036108199999999998 2024-04-24T00:37:13.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935150+0000", 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2756999999999999e-05 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935159+0000", 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935249+0000", 2024-04-24T00:37:13.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9470999999999998e-05 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935263+0000", 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4445e-05 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889988+0000", 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95472527200000001 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890286+0000", 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029795199999999998 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890670+0000", 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038387799999999999 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902981+0000", 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012310399 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:13.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903018+0000", 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903151+0000", 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013311700000000001 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903254+0000", 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010306699999999999 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111119 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.423281631000002, 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96712422899999995, 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111119 2024-04-24T00:37:13.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935744+0000", 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.206e-06 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935750+0000", 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.203e-06 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935761+0000", 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1266e-05 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935766+0000", 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2830000000000001e-06 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935841+0000", 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4685999999999996e-05 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935854+0000", 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3135e-05 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902400+0000", 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96654596800000003 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902638+0000", 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023839799999999999 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902868+0000", 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229084 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.437 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111159 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.394322746999997, 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.94163226300000002, 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.438 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111159 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964704+0000", 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964711+0000", 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4710000000000001e-06 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964727+0000", 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6416e-05 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964736+0000", 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9209999999999998e-06 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964809+0000", 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3047999999999996e-05 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964821+0000", 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1738e-05 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.905869+0000", 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.941048035 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906136+0000", 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026686400000000001 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906334+0000", 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019838500000000001 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.442 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:123125 3.1f 3:facc60b2:::rbd_data.115ee312a896.000000000000001b:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.685961597000002, 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118275349, 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 123125 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673065+0000", 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673074+0000", 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1500000000000005e-06 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673099+0000", 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692e-05 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673119+0000", 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9683000000000001e-05 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673252+0000", 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133354 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673490+0000", 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023785700000000001 2024-04-24T00:37:13.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.680775+0000", 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007284858 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791134+0000", 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110358923 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791167+0000", 2024-04-24T00:37:13.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2985999999999999e-05 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791260+0000", 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3060999999999996e-05 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791339+0000", 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8844999999999995e-05 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:123125 3.1f e168/163 ondisk, result = 0)", 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.660075841999998, 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092199987999999997, 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698951+0000", 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7460000000000001e-06 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698959+0000", 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.746e-06 2024-04-24T00:37:13.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698988+0000", 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9257000000000001e-05 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.699001+0000", 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3362e-05 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791078+0000", 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092076651999999995 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791120+0000", 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1789000000000003e-05 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791149+0000", 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9436000000000001e-05 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143625 3.18 3:1b6251e3:::rbd_data.115e8dbf22aa.0000000000000039:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.704380673999999, 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033012196000000001, 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143625 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654647+0000", 2024-04-24T00:37:13.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654660+0000", 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.309e-05 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654691+0000", 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0709e-05 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654717+0000", 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6267000000000001e-05 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654859+0000", 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014176400000000001 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.655117+0000", 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025819200000000002 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.656922+0000", 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00180528 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687423+0000", 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030501117000000001 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687502+0000", 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9127000000000003e-05 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687584+0000", 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2139999999999996e-05 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687656+0000", 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1989999999999993e-05 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143678 3.b 3:d70c067f:::rbd_data.115e8dbf22aa.000000000000005e:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.522092923999999, 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18713334100000001, 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143678 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836936+0000", 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9829999999999998e-06 2024-04-24T00:37:13.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836953+0000", 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7016000000000001e-05 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836986+0000", 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3319999999999999e-05 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837001+0000", 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:13.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837183+0000", 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018131700000000001 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837430+0000", 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024749599999999999 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:13.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.847946+0000", 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01051566 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023623+0000", 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17567688400000001 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023871+0000", 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024814800000000001 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023984+0000", 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011326499999999999 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024065+0000", 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1129000000000002e-05 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:143678 3.b e183/160 ondisk, result = 0)", 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.476686623999999, 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14174242100000001, 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882340+0000", 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882350+0000", 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882384+0000", 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3923e-05 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882407+0000", 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2619000000000001e-05 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023712+0000", 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.141305088 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023728+0000", 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.607e-05 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024081+0000", 2024-04-24T00:37:13.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352648 2024-04-24T00:37:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:13.469 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:13.470 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:13.467+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:13.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:13.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:13.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-24T00:37:13.572 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:13.570+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:13.668 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:13.668 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:13.668 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:13.672 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:13.667+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:13.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:13.694 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:13.695 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:13.712 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:13.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:13.730 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:13.730 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:13.748 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:13.748 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:13.773 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:13.770+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:13.786 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:13.822 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:13.823 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:13.841 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:13.842 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:13.847 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:13.847 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:13.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:13.869 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:13.869 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:13.889 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:13.883+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:13.926 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #37, image image_client.0-clone37 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent38 2024-04-24T00:37:13.974 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:13.970+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:13.984 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:13.984 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:14.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:14.089 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:14.087+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:14.168 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:14.169 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:14.176 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:14.171+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:14.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:14.246 INFO:teuthology.orchestra.run.smithi064.stdout:5805 writesame 0xc3db50c thru 0xc3e35cf data_size 0x6(0x80c4 bytes) 2024-04-24T00:37:14.277 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:14.274+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:14.280 INFO:teuthology.orchestra.run.smithi064.stdout:5807 read 0x30b73d0 thru 0x30bfc5c (0x888d bytes) 2024-04-24T00:37:14.286 INFO:teuthology.orchestra.run.smithi064.stdout:5808 trunc from 0xe96fe2c to 0xb5184d9 2024-04-24T00:37:14.312 INFO:teuthology.orchestra.run.smithi064.stdout:5809 clone 40 order 24 su 131072 sc 9 2024-04-24T00:37:14.349 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:14.349 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:14.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:14.378 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:14.374+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:14.479 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:14.474+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:14.521 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:14.521 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:14.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:14.581 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:14.579+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:14.664 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:14.664 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:14.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:14.681 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:14.678+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:14.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:37:14.782 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:14.778+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:14.850 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:14.851 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:14.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:14.883 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:14.878+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:14.984 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:14.978+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:14.989 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:14.989 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:15.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:15.085 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:15.083+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:15.134 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:15.135 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:15.148 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:15.186 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:15.183+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:15.273 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:15.274 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:15.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:15.288 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:15.282+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:15.388 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:15.383+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:15.453 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:15.453 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:15.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:15.513 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:15.510+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:15.584 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:15.585 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:15.590 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:15.587+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:15.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:15.691 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:15.687+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:15.755 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:37:15.760 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:15.760 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:15.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:15.821 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:15.818+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:15.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:15.892 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:15.886+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:15.954 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:15.954 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:15.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:16.017 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:16.014+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:16.050 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:16.050 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:16.067 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:16.067 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:16.084 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:16.084 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:16.093 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:16.090+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:16.100 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:16.120 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:16.120 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:16.126 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:16.126 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:16.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:16.141 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:16.141 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:16.160 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:16.160 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:16.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:16.181 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:16.194 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:16.191+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:16.295 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:16.290+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:16.307 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:16.307 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:16.307 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:16.307 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:16.308 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:16.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:16.409 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:16.406+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:16.488 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:16.488 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:16.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 177.95266541199999, 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.489 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:37:16.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:37:16.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:16.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 177.918628359, 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:16.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:37:16.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:37:16.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:37:16.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:37:16.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.80351436300001, 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.503 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:37:16.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:37:16.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:16.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.510 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.776064402, 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:37:16.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:37:16.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:16.515 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 98.855992569999998, 2024-04-24T00:37:16.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:37:16.516 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.516 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.516 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:16.516 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:37:16.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:37:16.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:37:16.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.520 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 98.849721312, 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.521 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:37:16.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:37:16.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:16.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:37:16.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:37:16.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:16.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:37:16.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:37:16.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:37:16.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:37:16.527 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.527 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.527 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.108965547, 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.528 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:37:16.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:37:16.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 26.088074000999999, 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:37:16.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:37:16.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162926 3.a 3:5391198c:::rbd_data.115e135fe8e7.0000000000000024:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7028172330000002, 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069942328999999998, 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162926 2024-04-24T00:37:16.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782103+0000", 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782107+0000", 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.895e-06 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782118+0000", 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1226999999999999e-05 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782128+0000", 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9909999999999997e-06 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.851774+0000", 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069645631 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.851886+0000", 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111928 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852045+0000", 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015879199999999999 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162938 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 16384~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.702778093, 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070154141000000003, 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.542 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162938 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0500000000000002e-07 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782146+0000", 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0489999999999997e-06 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782158+0000", 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1701e-05 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782169+0000", 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852078+0000", 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069909099000000002 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852143+0000", 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4906999999999994e-05 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852296+0000", 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015313699999999999 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.546 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162939 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 49152~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7027379959999998, 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070386700999999996, 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162939 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782186+0000", 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.924e-06 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782198+0000", 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2029e-05 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782208+0000", 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9240000000000002e-06 2024-04-24T00:37:16.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852330+0000", 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070121292000000002 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852410+0000", 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9945999999999997e-05 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852568+0000", 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158763 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162940 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.702698007, 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070748154999999993, 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.551 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162940 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:37:16.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782226+0000", 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782239+0000", 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2516e-05 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782249+0000", 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064000000000001e-05 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852602+0000", 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070352985000000007 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852685+0000", 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3584999999999993e-05 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852970+0000", 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284412 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.555 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162941 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 114688~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7026571740000001, 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071218216000000001, 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162941 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782263+0000", 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782267+0000", 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.172e-06 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782280+0000", 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2308e-05 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782290+0000", 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0674e-05 2024-04-24T00:37:16.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853061+0000", 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070770437000000005 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853196+0000", 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013527100000000001 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853481+0000", 2024-04-24T00:37:16.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028453700000000002 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162942 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 147456~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7026156700000001, 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071662612000000001, 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.560 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162942 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:37:16.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782305+0000", 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3399999999999998e-07 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782308+0000", 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5480000000000002e-06 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782321+0000", 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2814e-05 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782331+0000", 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0363999999999999e-05 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853550+0000", 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071219062999999999 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853680+0000", 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129219 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853967+0000", 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028687 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162943 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 180224~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.564 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7025745059999999, 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071922708000000002, 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162943 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782346+0000", 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6300000000000004e-07 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782350+0000", 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9129999999999996e-06 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782361+0000", 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1249e-05 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782370+0000", 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0829999999999993e-06 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854021+0000", 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071651144 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854119+0000", 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7157999999999994e-05 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854268+0000", 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014929799999999999 2024-04-24T00:37:16.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162944 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 212992~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7025367610000002, 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072204638000000002, 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162944 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782384+0000", 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7800000000000001e-07 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782387+0000", 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7720000000000001e-06 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782399+0000", 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1508000000000001e-05 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782409+0000", 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7550000000000006e-06 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854321+0000", 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071912767000000002 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854437+0000", 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115282 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854587+0000", 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015087600000000001 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162945 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 245760~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.573 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7024979770000002, 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072460953999999994, 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162945 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:37:16.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782427+0000", 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6210000000000003e-06 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782447+0000", 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0129999999999999e-05 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782457+0000", 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.03e-05 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854619+0000", 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072161259000000005 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854739+0000", 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120542 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854883+0000", 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143354 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.577 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162946 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 278528~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.702448994, 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072673115999999996, 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.578 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162946 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782493+0000", 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1645999999999999e-05 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782503+0000", 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4509999999999998e-06 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782511+0000", 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854939+0000", 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072427668000000001 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855054+0000", 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114957 2024-04-24T00:37:16.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855144+0000", 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0175999999999999e-05 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162947 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 311296~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.702399185, 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072916576999999996, 2024-04-24T00:37:16.582 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162947 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782521+0000", 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782524+0000", 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.971e-06 2024-04-24T00:37:16.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782532+0000", 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7940000000000003e-06 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782538+0000", 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855191+0000", 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072653180999999997 2024-04-24T00:37:16.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855292+0000", 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100948 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855437+0000", 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014472700000000001 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162948 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.7023479740000003, 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073391995000000002, 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162948 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:16.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782573+0000", 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9700000000000005e-07 2024-04-24T00:37:16.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782577+0000", 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782587+0000", 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5140000000000007e-06 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782594+0000", 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.968e-06 2024-04-24T00:37:16.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855774+0000", 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073180178999999998 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855862+0000", 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7793999999999993e-05 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:16.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855964+0000", 2024-04-24T00:37:16.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010183099999999999 2024-04-24T00:37:16.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.591 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:16.591 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:16.591 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:16.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:16.593 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:16.590+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:16.693 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:16.691+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:16.768 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:16.768 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:16.768 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:16.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:16.829 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:16.826+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:16.894 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:16.891+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:16.897 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:16.898 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:16.898 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:16.898 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:16.898 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:16.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:16.994 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #38, image image_client.0-clone38 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent39 2024-04-24T00:37:17.001 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:16.998+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 216.16347135699999, 2024-04-24T00:37:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:37:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:37:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 183.231505669, 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:37:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 183.23025692100001, 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:37:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.072632817, 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:37:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:37:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:37:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.05913827399999, 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.797357159, 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:37:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:37:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:37:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:37:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:37:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:37:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:37:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:37:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:37:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:37:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:37:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:37:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:37:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:37:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:37:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:37:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:37:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:37:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:37:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:17.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:37:17.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:37:17.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:37:17.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:37:17.135 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.135 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.135 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.135 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:17.136 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:17.136 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.54377101599999, 2024-04-24T00:37:17.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:37:17.136 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.136 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.136 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.137 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:37:17.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.137 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:17.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:37:17.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.51609715199999, 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.143 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:37:17.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.02207433199999, 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:37:17.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:37:17.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:37:17.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.013065498, 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:37:17.155 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.159 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.51136654999999, 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:17.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:37:17.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.511304204, 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.164 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:37:17.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.51079452499999, 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.168 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:17.172 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.51068346299999, 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:17.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.176 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.51061819700001, 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:37:17.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:37:17.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:37:17.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.51055813100001, 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:37:17.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:17.185 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.51050143800001, 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.51043210500001, 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.190 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.51034148599999, 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.194 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.510298073, 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:17.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:17.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:37:17.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:17.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:37:17.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:37:17.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:17.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:17.203 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:17.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:17.205 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:17.199+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:17.305 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:17.303+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:17.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:17.449 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:17.447+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:17.502 INFO:teuthology.orchestra.run.smithi064.stdout:5810 write 0x485286f thru 0x485d496 (0xac28 bytes) 2024-04-24T00:37:17.519 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:17.514+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:17.525 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:17.526 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:17.526 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:17.526 INFO:teuthology.orchestra.run.smithi064.stdout:5811 trunc from 0xb5184d9 to 0x8067c70 2024-04-24T00:37:17.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:17.560 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.582 INFO:teuthology.orchestra.run.smithi064.stdout:5813 write 0xd6d38f4 thru 0xd6dd2e1 (0x99ee bytes) 2024-04-24T00:37:17.590 INFO:teuthology.orchestra.run.smithi064.stdout:5814 trunc from 0xd6dd2e2 to 0xccef125 2024-04-24T00:37:17.604 INFO:teuthology.orchestra.run.smithi064.stdout:5815 write 0x44e9fec thru 0x44f9294 (0xf2a9 bytes) 2024-04-24T00:37:17.615 INFO:teuthology.orchestra.run.smithi064.stdout:5817 punch from 0xc34ac2c to 0xc35988c, (0xec60 bytes) 2024-04-24T00:37:17.616 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.616 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.616 INFO:teuthology.orchestra.run.smithi064.stdout:5821 punch from 0xc8e72ca to 0xc8ee612, (0x7348 bytes) 2024-04-24T00:37:17.616 INFO:teuthology.orchestra.run.smithi064.stdout:5822 write 0xef250 thru 0xf023b (0xfec bytes) 2024-04-24T00:37:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:5824 punch from 0x19c3d85 to 0x19ccbf7, (0x8e72 bytes) 2024-04-24T00:37:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:5825 read 0x2af2631 thru 0x2afedd6 (0xc7a6 bytes) 2024-04-24T00:37:17.620 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:17.615+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:17.620 INFO:teuthology.orchestra.run.smithi064.stdout:5826 read 0x44e6665 thru 0x44f4af3 (0xe48f bytes) 2024-04-24T00:37:17.622 INFO:teuthology.orchestra.run.smithi064.stdout:5827 read 0xab31a6c thru 0xab40bb2 (0xf147 bytes) 2024-04-24T00:37:17.623 INFO:teuthology.orchestra.run.smithi064.stdout:5829 writesame 0x3dfdb0 thru 0x3e6a73 data_size 0x2(0x6cc4 bytes) 2024-04-24T00:37:17.687 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110941 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 32768~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.800488837000003, 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99518982899999997, 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.688 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110941 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882159+0000", 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882161+0000", 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882169+0000", 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8480000000000001e-06 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882173+0000", 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6110000000000004e-06 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886136+0000", 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039630810000000002 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886147+0000", 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0484e-05 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876102+0000", 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98995489400000003 2024-04-24T00:37:17.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876434+0000", 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033270799999999997 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.877348+0000", 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091357700000000005 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110942 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 229376~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.800467091000002, 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99714419399999998, 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110942 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.694 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:17.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5199999999999998e-07 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882182+0000", 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8819999999999999e-06 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882190+0000", 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3629999999999996e-06 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882194+0000", 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5000000000000001e-06 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886176+0000", 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039815329999999998 2024-04-24T00:37:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886184+0000", 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878386+0000", 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99220202999999996 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878535+0000", 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001494 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879324+0000", 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078909599999999998 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110958 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 65536~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.800370778999998, 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.998192368, 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110958 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882277+0000", 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882278+0000", 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6419999999999999e-06 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882284+0000", 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3890000000000004e-06 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882288+0000", 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.138e-06 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886955+0000", 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046673289999999996 2024-04-24T00:37:17.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886964+0000", 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8310000000000007e-06 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879623+0000", 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99265892499999997 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879818+0000", 2024-04-24T00:37:17.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019486800000000001 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880469+0000", 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065093099999999999 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110959 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 262144~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.800310881000001, 2024-04-24T00:37:17.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99945926699999998, 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110959 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882337+0000", 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4499999999999997e-07 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882340+0000", 2024-04-24T00:37:17.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0259999999999999e-06 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882358+0000", 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8689999999999999e-05 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882365+0000", 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3879999999999997e-06 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886989+0000", 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046244930000000004 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886997+0000", 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880603+0000", 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99360585199999996 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880779+0000", 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175792 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881795+0000", 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001016922 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110974 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 98304~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.800163435999998, 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.000617986, 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110974 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2100000000000002e-07 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882486+0000", 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.215e-06 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882494+0000", 2024-04-24T00:37:17.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9410000000000007e-06 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882500+0000", 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2389999999999999e-06 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888427+0000", 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059265029999999996 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888436+0000", 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2019999999999993e-06 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881936+0000", 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99350027399999996 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.882146+0000", 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020958500000000001 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883102+0000", 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095560600000000001 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110975 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 294912~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.800139954000002, 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002014129, 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.716 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110975 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882508+0000", 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882510+0000", 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.148e-06 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882515+0000", 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7289999999999997e-06 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882520+0000", 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3290000000000004e-06 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888463+0000", 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059433669999999997 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888470+0000", 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.187e-06 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883245+0000", 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99477520799999997 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883497+0000", 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025132699999999997 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884521+0000", 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001024435 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110990 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 131072~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.799988816000003, 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0028161980000001, 2024-04-24T00:37:17.721 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110990 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882659+0000", 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4299999999999998e-07 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882661+0000", 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-06 2024-04-24T00:37:17.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882667+0000", 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.879e-06 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882671+0000", 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5349999999999998e-06 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889926+0000", 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072543970000000001 2024-04-24T00:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889935+0000", 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7070000000000004e-06 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884669+0000", 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99473391700000002 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884841+0000", 2024-04-24T00:37:17.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017212600000000001 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885474+0000", 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063308099999999996 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110991 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 327680~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.799969763, 2024-04-24T00:37:17.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0041004259999999, 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110991 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.727 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:17.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882678+0000", 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882684+0000", 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3219999999999998e-06 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882691+0000", 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882705+0000", 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3813e-05 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889972+0000", 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072669309999999999 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889981+0000", 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3059999999999998e-06 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885568+0000", 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99558754299999996 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885798+0000", 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023032799999999999 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886778+0000", 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097934600000000008 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111006 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 163840~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.799833970999998, 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0048379629999999, 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111006 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882815+0000", 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7549999999999999e-06 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882822+0000", 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0539999999999997e-06 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882827+0000", 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3619999999999999e-06 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890715+0000", 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078878809999999994 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890723+0000", 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6289999999999999e-06 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886854+0000", 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99613063000000002 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887032+0000", 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017786200000000001 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887651+0000", 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061942900000000001 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111007 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 360448~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.799814542, 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061255339999999, 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111007 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0400000000000002e-07 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882835+0000", 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7129999999999999e-06 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882842+0000", 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1280000000000002e-06 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882846+0000", 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5600000000000004e-06 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890749+0000", 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079023110000000004 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890758+0000", 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2070000000000001e-06 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887756+0000", 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99699837099999999 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888019+0000", 2024-04-24T00:37:17.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263014 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888958+0000", 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093882600000000003 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111032 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.769880786000002, 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97659819800000003, 2024-04-24T00:37:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111032 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912767+0000", 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912769+0000", 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6759999999999999e-06 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912777+0000", 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7659999999999998e-06 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912782+0000", 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0590000000000002e-06 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913091+0000", 2024-04-24T00:37:17.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030847099999999999 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913101+0000", 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889077+0000", 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.975975437 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889228+0000", 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015131699999999999 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889364+0000", 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013657900000000001 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111065 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.748 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.759085190999997, 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96634889000000002, 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111065 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923563+0000", 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7100000000000001e-07 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923566+0000", 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.658e-06 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923577+0000", 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268999999999999e-05 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923588+0000", 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1299e-05 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923668+0000", 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0012000000000006e-05 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923678+0000", 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9019999999999994e-06 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889478+0000", 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96580058899999999 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889697+0000", 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218285 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889911+0000", 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000214105 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111116 3.1f 3:f85dfb34:::rbd_data.115e397a9cd4.000000000000000c:head [write 0~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.753 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.747883749000003, 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96849046100000002, 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111116 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:17.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934766+0000", 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3580000000000001e-06 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935127+0000", 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036108199999999998 2024-04-24T00:37:17.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935150+0000", 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2756999999999999e-05 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935159+0000", 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935249+0000", 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9470999999999998e-05 2024-04-24T00:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935263+0000", 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4445e-05 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889988+0000", 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95472527200000001 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890286+0000", 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029795199999999998 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890670+0000", 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038387799999999999 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902981+0000", 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012310399 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903018+0000", 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:37:17.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903151+0000", 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013311700000000001 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903254+0000", 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010306699999999999 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111119 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.746903803999999, 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96712422899999995, 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111119 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935744+0000", 2024-04-24T00:37:17.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.206e-06 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935750+0000", 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.203e-06 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935761+0000", 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1266e-05 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935766+0000", 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2830000000000001e-06 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935841+0000", 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4685999999999996e-05 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935854+0000", 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3135e-05 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902400+0000", 2024-04-24T00:37:17.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96654596800000003 2024-04-24T00:37:17.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902638+0000", 2024-04-24T00:37:17.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023839799999999999 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902868+0000", 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229084 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111159 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:17.766 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.717944920000001, 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.94163226300000002, 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111159 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:17.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964704+0000", 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964711+0000", 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4710000000000001e-06 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964727+0000", 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6416e-05 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964736+0000", 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9209999999999998e-06 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964809+0000", 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3047999999999996e-05 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964821+0000", 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1738e-05 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.905869+0000", 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.941048035 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906136+0000", 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026686400000000001 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906334+0000", 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019838500000000001 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:123125 3.1f 3:facc60b2:::rbd_data.115ee312a896.000000000000001b:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-24T00:37:17.771 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.009583769999999, 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118275349, 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 123125 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:17.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673065+0000", 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673074+0000", 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1500000000000005e-06 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673099+0000", 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692e-05 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673119+0000", 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9683000000000001e-05 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673252+0000", 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133354 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673490+0000", 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023785700000000001 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.680775+0000", 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007284858 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791134+0000", 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110358923 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791167+0000", 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2985999999999999e-05 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791260+0000", 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3060999999999996e-05 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791339+0000", 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8844999999999995e-05 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:123125 3.1f e168/163 ondisk, result = 0)", 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.983698015000002, 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092199987999999997, 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.777 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698951+0000", 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7460000000000001e-06 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698959+0000", 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.746e-06 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698988+0000", 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9257000000000001e-05 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.699001+0000", 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3362e-05 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791078+0000", 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092076651999999995 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791120+0000", 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1789000000000003e-05 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791149+0000", 2024-04-24T00:37:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9436000000000001e-05 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143625 3.18 3:1b6251e3:::rbd_data.115e8dbf22aa.0000000000000039:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.028002847, 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033012196000000001, 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143625 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654647+0000", 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654660+0000", 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.309e-05 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654691+0000", 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0709e-05 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654717+0000", 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6267000000000001e-05 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654859+0000", 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014176400000000001 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.655117+0000", 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025819200000000002 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.656922+0000", 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00180528 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687423+0000", 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030501117000000001 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687502+0000", 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9127000000000003e-05 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687584+0000", 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2139999999999996e-05 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687656+0000", 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1989999999999993e-05 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143678 3.b 3:d70c067f:::rbd_data.115e8dbf22aa.000000000000005e:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:17.786 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.845715096999999, 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18713334100000001, 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143678 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:17.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836936+0000", 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9829999999999998e-06 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836953+0000", 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7016000000000001e-05 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836986+0000", 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3319999999999999e-05 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837001+0000", 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837183+0000", 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018131700000000001 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837430+0000", 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024749599999999999 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.847946+0000", 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01051566 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023623+0000", 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17567688400000001 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023871+0000", 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024814800000000001 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023984+0000", 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011326499999999999 2024-04-24T00:37:17.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024065+0000", 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1129000000000002e-05 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:143678 3.b e183/160 ondisk, result = 0)", 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.800308797, 2024-04-24T00:37:17.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14174242100000001, 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:17.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882340+0000", 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882350+0000", 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:17.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882384+0000", 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3923e-05 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882407+0000", 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2619000000000001e-05 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023712+0000", 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.141305088 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023728+0000", 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.607e-05 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024081+0000", 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352648 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:17.796 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:17.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:37:17.798 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:17.794+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:17.798 INFO:teuthology.orchestra.run.smithi064.stdout:5830 write 0x8833e07 thru 0x8836d34 (0x2f2e bytes) 2024-04-24T00:37:17.798 INFO:teuthology.orchestra.run.smithi064.stdout:5831 write 0x2ad3eab thru 0x2ad8004 (0x415a bytes) 2024-04-24T00:37:17.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:5832 read 0x3274d71 thru 0x327c771 (0x7a01 bytes) 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:5837 trunc from 0xccef125 to 0xc634ba8 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:5839 trunc from 0xc634ba8 to 0x59972ce 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:5841 punch from 0x25a87e2 to 0x25b10b0, (0x88ce bytes) 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:5842 read 0x1420f1f thru 0x1426ed3 (0x5fb5 bytes) 2024-04-24T00:37:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:5844 write 0xabb597b thru 0xabbb7ac (0x5e32 bytes) 2024-04-24T00:37:17.813 INFO:teuthology.orchestra.run.smithi064.stdout:5845 trunc from 0xabbb7ad to 0xc0f2625 2024-04-24T00:37:17.825 INFO:teuthology.orchestra.run.smithi064.stdout:5846 punch from 0x6e4f68f to 0x6e5ab27, (0xb498 bytes) 2024-04-24T00:37:17.829 INFO:teuthology.orchestra.run.smithi064.stdout:5847 trunc from 0xc0f2625 to 0x9fed621 2024-04-24T00:37:17.845 INFO:teuthology.orchestra.run.smithi064.stdout:5848 punch from 0x273cd7 to 0x282e3d, (0xf166 bytes) 2024-04-24T00:37:17.845 INFO:teuthology.orchestra.run.smithi064.stdout:5849 writesame 0x8146e12 thru 0x814a69f data_size 0x39(0x388e bytes) 2024-04-24T00:37:17.867 INFO:teuthology.orchestra.run.smithi064.stdout:5850 write 0xd636926 thru 0xd63d872 (0x6f4d bytes) 2024-04-24T00:37:17.888 INFO:teuthology.orchestra.run.smithi064.stdout:5852 write 0xe145b78 thru 0xe155afb (0xff84 bytes) 2024-04-24T00:37:17.897 INFO:teuthology.orchestra.run.smithi064.stdout:5853 writesame 0xde43f7b thru 0xde4fd3f data_size 0x25(0xbdc5 bytes) 2024-04-24T00:37:17.898 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:17.894+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:17.918 INFO:teuthology.orchestra.run.smithi064.stdout:5854 write 0xe870f4f thru 0xe87c844 (0xb8f6 bytes) 2024-04-24T00:37:17.923 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.923 INFO:teuthology.orchestra.run.smithi064.stdout:5857 write 0x45a8077 thru 0x45a85be (0x548 bytes) 2024-04-24T00:37:17.926 INFO:teuthology.orchestra.run.smithi064.stdout:5858 write 0x248b97 thru 0x2540e6 (0xb550 bytes) 2024-04-24T00:37:17.929 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:17.930 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:17.937 INFO:teuthology.orchestra.run.smithi064.stdout:5859 read 0x1404d34 thru 0x1407ba8 (0x2e75 bytes) 2024-04-24T00:37:17.937 INFO:teuthology.orchestra.run.smithi064.stdout:5860 read 0x29ce8c4 thru 0x29d5a91 (0x71ce bytes) 2024-04-24T00:37:17.938 INFO:teuthology.orchestra.run.smithi064.stdout:5861 read 0xbb051 thru 0xbb0df (0x8f bytes) 2024-04-24T00:37:17.940 INFO:teuthology.orchestra.run.smithi064.stdout:5862 write 0xa45a09f thru 0xa468e5e (0xedc0 bytes) 2024-04-24T00:37:17.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:17.950 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:17.951 INFO:teuthology.orchestra.run.smithi064.stdout:5864 punch from 0x152a241 to 0x1532889, (0x8648 bytes) 2024-04-24T00:37:17.951 INFO:teuthology.orchestra.run.smithi064.stdout:5866 writesame 0xdb4945c thru 0xdb56656 data_size 0x5(0xd1fb bytes) 2024-04-24T00:37:17.978 INFO:teuthology.orchestra.run.smithi064.stdout:5868 write 0xddd1dee thru 0xddd6301 (0x4514 bytes) 2024-04-24T00:37:17.981 INFO:teuthology.orchestra.run.smithi064.stdout:5871 writesame 0xdf9b103 thru 0xdfaa5ff data_size 0x1(0xf4fd bytes) 2024-04-24T00:37:17.999 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:17.995+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:18.071 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:18.071 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:18.071 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:18.072 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:18.072 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:18.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:18.129 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:18.127+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:18.131 INFO:teuthology.orchestra.run.smithi064.stdout:5872 punch from 0x9073c1b to 0x908380f, (0xfbf4 bytes) 2024-04-24T00:37:18.131 INFO:teuthology.orchestra.run.smithi064.stdout:5873 read 0x1517a26 thru 0x152535a (0xd935 bytes) 2024-04-24T00:37:18.132 INFO:teuthology.orchestra.run.smithi064.stdout:5874 writesame 0x6522b4 thru 0x661412 data_size 0x2b(0xf15f bytes) 2024-04-24T00:37:18.136 INFO:teuthology.orchestra.run.smithi064.stdout:5875 punch from 0x21ae99 to 0x227b29, (0xcc90 bytes) 2024-04-24T00:37:18.137 INFO:teuthology.orchestra.run.smithi064.stdout:5876 write 0xcf62936 thru 0xcf6e0d0 (0xb79b bytes) 2024-04-24T00:37:18.142 INFO:teuthology.orchestra.run.smithi064.stdout:5877 read 0xd35d033 thru 0xd3648d8 (0x78a6 bytes) 2024-04-24T00:37:18.142 INFO:teuthology.orchestra.run.smithi064.stdout:5879 write 0x907ae23 thru 0x908527d (0xa45b bytes) 2024-04-24T00:37:18.151 INFO:teuthology.orchestra.run.smithi064.stdout:5880 write 0x83de37 thru 0x84cf81 (0xf14b bytes) 2024-04-24T00:37:18.160 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:18.160 INFO:teuthology.orchestra.run.smithi064.stdout:5882 write 0x31b2876 thru 0x31bfcc6 (0xd451 bytes) 2024-04-24T00:37:18.179 INFO:teuthology.orchestra.run.smithi064.stdout:5884 trunc from 0xe87c845 to 0x7e1a979 2024-04-24T00:37:18.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:18.201 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:18.195+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:18.211 INFO:teuthology.orchestra.run.smithi064.stdout:5885 clone 41 order 22 su 65536 sc 3 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.823697987, 2024-04-24T00:37:18.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:37:18.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.268 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.768186501, 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:37:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:37:18.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 151.66760985299999, 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:37:18.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:37:18.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:37:18.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:37:18.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:37:18.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:37:18.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:37:18.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:37:18.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:37:18.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:37:18.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:37:18.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.282 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.282 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:37:18.283 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:18.283 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 151.62604489899999, 2024-04-24T00:37:18.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:37:18.283 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.283 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.284 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.284 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:18.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:37:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:37:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:37:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:37:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:37:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:37:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:37:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:37:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:37:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:37:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:37:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:37:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:37:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:37:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:37:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:18.294 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 114.06231333700001, 2024-04-24T00:37:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:37:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.295 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:37:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:37:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:37:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:37:18.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:37:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:37:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:37:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:37:18.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:37:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:37:18.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:37:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:37:18.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:37:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:37:18.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:37:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:37:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:37:18.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:18.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:37:18.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:37:18.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:18.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:37:18.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:37:18.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:37:18.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:37:18.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.309 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:37:18.309 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:18.309 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 110.55933994, 2024-04-24T00:37:18.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:37:18.309 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:37:18.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.311 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:37:18.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.311 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:18.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:18.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:37:18.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:37:18.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:37:18.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:37:18.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:37:18.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:37:18.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:37:18.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:37:18.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:37:18.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:37:18.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:37:18.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:37:18.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:37:18.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:37:18.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:37:18.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:18.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:37:18.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:37:18.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:37:18.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:37:18.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:37:18.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:37:18.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.321025707, 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:18.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:18.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:18.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:37:18.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:37:18.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:37:18.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:18.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:37:18.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:37:18.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:37:18.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:37:18.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.330 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.331 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:18.331 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.28608528399999, 2024-04-24T00:37:18.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:37:18.331 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.331 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.331 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.332 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:37:18.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.332 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:18.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:37:18.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:37:18.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:37:18.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:37:18.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:37:18.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:37:18.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:37:18.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:37:18.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:37:18.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:37:18.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:37:18.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:37:18.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:37:18.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:18.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.286064411, 2024-04-24T00:37:18.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:37:18.340 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.340 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.340 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:37:18.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.341 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:18.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:18.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:37:18.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.286041032, 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:37:18.345 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.28601949, 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.351 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.355 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.28599797, 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:37:18.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:37:18.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.285951813, 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:37:18.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:37:18.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:37:18.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.28592769399999, 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:37:18.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.28590429, 2024-04-24T00:37:18.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:18.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:37:18.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.28587651700001, 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:37:18.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.28585510400001, 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.285834513, 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:37:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.069764270999997, 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:37:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:37:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.041793366999997, 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:18.396 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:18.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:18.398 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:18.399 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:18.399 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:18.400 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:18.400 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:18.400 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:18.400 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:18.395+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:18.416 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:18.416 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:18.437 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:18.437 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:18.457 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:18.458 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:18.478 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:18.479 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:18.499 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:18.494+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:18.546 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:18.547 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:18.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:18.600 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:18.594+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:18.685 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:18.685 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:18.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:18.700 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:18.695+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:18.765 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:18.765 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:18.765 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-24T00:37:18.765 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:37:18.766 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:37:18.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-04-24T00:37:18.769 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:37:18.813 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:18.810+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:18.835 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:18.835 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:18.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:18.899 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:18.899 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "" 2024-04-24T00:37:18.899 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:18.903 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:18.898+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:19.003 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:18.998+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:19.020 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:19.020 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:19.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:19.104 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:19.099+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:19.162 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:19.162 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:19.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:19.217 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:19.215+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:19.306 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:19.303+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:19.346 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:19.346 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:19.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:19.409 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:19.406+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:19.495 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:19.495 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:19.508 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:19.502+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:19.509 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:19.609 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:19.606+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:19.710 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:19.706+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:19.724 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:19.724 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:19.744 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:19.833 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:19.830+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:19.911 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:19.907+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:19.920 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:19.920 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:19.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:20.021 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:20.018+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:20.099 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:20.099 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:20.114 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:20.114 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:20.110+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:20.214 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:20.210+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:20.280 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:20.281 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:20.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:20.337 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:20.334+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:20.416 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:20.410+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:20.462 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:20.462 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:20.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:20.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:20.521 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:20.518+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:20.618 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:20.614+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:20.646 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:20.646 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:20.659 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:20.660 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:20.661 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:20.676 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:20.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:20.693 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:20.693 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:20.709 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:20.710 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:20.728 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:20.728 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:20.752 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:20.750+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:20.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:37:20.834 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:20.835 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:20.845 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:20.843+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:20.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:20.921 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:20.919+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:20.945 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #39, image image_client.0-clone39 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent40 2024-04-24T00:37:20.953 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:20.954 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:20.974 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:20.974 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:20.993 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:20.993 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:21.022 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:21.024 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:21.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:21.024 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:21.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.025 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:37:21.025 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:21.025 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 182.48673910900001, 2024-04-24T00:37:21.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:37:21.025 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.025 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.026 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.026 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.026 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.026 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.026 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:37:21.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.027 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:21.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:37:21.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:37:21.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:37:21.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:37:21.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:21.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:21.033 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 182.45270205599999, 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:37:21.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.33758806, 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:37:21.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.043 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.310138099, 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:37:21.047 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 103.39006626699999, 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:37:21.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:37:21.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:37:21.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 103.383795009, 2024-04-24T00:37:21.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:37:21.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:37:21.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:37:21.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 30.643039244000001, 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.058 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 30.622147697999999, 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.064 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:37:21.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162926 3.a 3:5391198c:::rbd_data.115e135fe8e7.0000000000000024:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2368909299999995, 2024-04-24T00:37:21.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069942328999999998, 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162926 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782103+0000", 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782107+0000", 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.895e-06 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782118+0000", 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1226999999999999e-05 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782128+0000", 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9909999999999997e-06 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.851774+0000", 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069645631 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.851886+0000", 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111928 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852045+0000", 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015879199999999999 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162938 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 16384~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2368517899999993, 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070154141000000003, 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.073 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162938 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:37:21.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0500000000000002e-07 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782146+0000", 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0489999999999997e-06 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782158+0000", 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1701e-05 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782169+0000", 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852078+0000", 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069909099000000002 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852143+0000", 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4906999999999994e-05 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852296+0000", 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015313699999999999 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.077 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162939 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 49152~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2368116929999999, 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070386700999999996, 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162939 2024-04-24T00:37:21.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782186+0000", 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.924e-06 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782198+0000", 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2029e-05 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782208+0000", 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9240000000000002e-06 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852330+0000", 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070121292000000002 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852410+0000", 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9945999999999997e-05 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852568+0000", 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158763 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162940 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2367717040000006, 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070748154999999993, 2024-04-24T00:37:21.082 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162940 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782226+0000", 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:37:21.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782239+0000", 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2516e-05 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782249+0000", 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064000000000001e-05 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852602+0000", 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070352985000000007 2024-04-24T00:37:21.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852685+0000", 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3584999999999993e-05 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852970+0000", 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284412 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162941 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 114688~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2367308710000007, 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071218216000000001, 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162941 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782263+0000", 2024-04-24T00:37:21.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782267+0000", 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.172e-06 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782280+0000", 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2308e-05 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782290+0000", 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0674e-05 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853061+0000", 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070770437000000005 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853196+0000", 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013527100000000001 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853481+0000", 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028453700000000002 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162942 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 147456~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2366893670000003, 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071662612000000001, 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.091 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162942 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782305+0000", 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3399999999999998e-07 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782308+0000", 2024-04-24T00:37:21.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5480000000000002e-06 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782321+0000", 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2814e-05 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782331+0000", 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0363999999999999e-05 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853550+0000", 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071219062999999999 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853680+0000", 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129219 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853967+0000", 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028687 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162943 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 180224~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2366482029999997, 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071922708000000002, 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162943 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.096 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782346+0000", 2024-04-24T00:37:21.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6300000000000004e-07 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782350+0000", 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9129999999999996e-06 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782361+0000", 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1249e-05 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782370+0000", 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0829999999999993e-06 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854021+0000", 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071651144 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854119+0000", 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7157999999999994e-05 2024-04-24T00:37:21.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854268+0000", 2024-04-24T00:37:21.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014929799999999999 2024-04-24T00:37:21.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.100 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162944 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 212992~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2366104579999995, 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072204638000000002, 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.101 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162944 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782384+0000", 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7800000000000001e-07 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782387+0000", 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7720000000000001e-06 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782399+0000", 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1508000000000001e-05 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782409+0000", 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7550000000000006e-06 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854321+0000", 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071912767000000002 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854437+0000", 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115282 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854587+0000", 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015087600000000001 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.105 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162945 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 245760~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2365716740000003, 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072460953999999994, 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162945 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782427+0000", 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6210000000000003e-06 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782447+0000", 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0129999999999999e-05 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782457+0000", 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.03e-05 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854619+0000", 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072161259000000005 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854739+0000", 2024-04-24T00:37:21.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120542 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854883+0000", 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143354 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162946 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 278528~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:37:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2365226909999993, 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072673115999999996, 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162946 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782493+0000", 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1645999999999999e-05 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782503+0000", 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4509999999999998e-06 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782511+0000", 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:37:21.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854939+0000", 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072427668000000001 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855054+0000", 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114957 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855144+0000", 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0175999999999999e-05 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162947 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 311296~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2364728819999993, 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072916576999999996, 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.115 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162947 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782521+0000", 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782524+0000", 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.971e-06 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782532+0000", 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7940000000000003e-06 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782538+0000", 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855191+0000", 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072653180999999997 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855292+0000", 2024-04-24T00:37:21.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100948 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855437+0000", 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014472700000000001 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162948 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 8.2364216710000004, 2024-04-24T00:37:21.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073391995000000002, 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162948 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782573+0000", 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9700000000000005e-07 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782577+0000", 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-04-24T00:37:21.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782587+0000", 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5140000000000007e-06 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782594+0000", 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.968e-06 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855774+0000", 2024-04-24T00:37:21.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073180178999999998 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855862+0000", 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7793999999999993e-05 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855964+0000", 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010183099999999999 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.124 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.124 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:21.124 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:21.055+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:21.125 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:21.126 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:21.123+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:21.226 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:21.223+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:21.252 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:21.252 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:21.253 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:21.253 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:21.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:21.320 INFO:teuthology.orchestra.run.smithi064.stdout:5886 write 0x2e311d0 thru 0x2e39ab3 (0x88e4 bytes) 2024-04-24T00:37:21.327 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:21.323+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:21.355 INFO:teuthology.orchestra.run.smithi064.stdout:5887 write 0xd39dfbf thru 0xd3a0816 (0x2858 bytes) 2024-04-24T00:37:21.362 INFO:teuthology.orchestra.run.smithi064.stdout:5888 read 0x4cae753 thru 0x4cb4942 (0x61f0 bytes) 2024-04-24T00:37:21.378 INFO:teuthology.orchestra.run.smithi064.stdout:5889 write 0xe3b5a13 thru 0xe3bfb69 (0xa157 bytes) 2024-04-24T00:37:21.386 INFO:teuthology.orchestra.run.smithi064.stdout:5890 read 0x7ef9a5d thru 0x7f01eca (0x846e bytes) 2024-04-24T00:37:21.387 INFO:teuthology.orchestra.run.smithi064.stdout:5891 writesame 0xa23d9b5 thru 0xa246f8c data_size 0x89(0x95d8 bytes) 2024-04-24T00:37:21.394 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:21.394 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:21.394 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:21.394 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:21.395 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:21.396 INFO:teuthology.orchestra.run.smithi064.stdout:5893 read 0x6979ff thru 0x69bbab (0x41ad bytes) 2024-04-24T00:37:21.405 INFO:teuthology.orchestra.run.smithi064.stdout:5895 trunc from 0xe3bfb6a to 0xa28bb97 2024-04-24T00:37:21.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:21.428 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:21.423+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:21.430 INFO:teuthology.orchestra.run.smithi064.stdout:5901 read 0x9610659 thru 0x961b994 (0xb33c bytes) 2024-04-24T00:37:21.430 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:21.442 INFO:teuthology.orchestra.run.smithi064.stdout:5903 writesame 0xa84192e thru 0xa844773 data_size 0x1(0x2e46 bytes) 2024-04-24T00:37:21.461 INFO:teuthology.orchestra.run.smithi064.stdout:5904 trunc from 0xa844774 to 0x2490bd0 2024-04-24T00:37:21.497 INFO:teuthology.orchestra.run.smithi064.stdout:5905 read 0x1e2d7d9 thru 0x1e394cf (0xbcf7 bytes) 2024-04-24T00:37:21.499 INFO:teuthology.orchestra.run.smithi064.stdout:5907 trunc from 0x2490bd0 to 0x1a374b2 2024-04-24T00:37:21.529 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:21.526+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:21.538 INFO:teuthology.orchestra.run.smithi064.stdout:5909 read 0x13a3419 thru 0x13ad9ab (0xa593 bytes) 2024-04-24T00:37:21.544 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:21.545 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:21.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:21.545 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:21.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.545 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:37:21.546 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:21.546 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.61615416399999, 2024-04-24T00:37:21.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:37:21.546 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.546 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.547 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.547 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.547 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:37:21.547 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:37:21.547 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:37:21.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.547 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:21.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:21.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:37:21.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:37:21.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:37:21.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:37:21.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:37:21.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:37:21.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:37:21.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:37:21.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:37:21.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:37:21.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:21.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:37:21.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:37:21.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:37:21.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:21.556 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.684188476, 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:37:21.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 187.68293972800001, 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.563 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:37:21.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:37:21.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.569 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.525315624, 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:37:21.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:37:21.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:37:21.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:37:21.575 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.51182108099999, 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:21.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.579 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.250039966, 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:37:21.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:37:21.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 164.996453823, 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:37:21.586 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:37:21.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:37:21.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:21.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 164.96877995899999, 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:21.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:21.597 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.47475713899999, 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.598 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:37:21.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:37:21.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:37:21.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:37:21.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.46574830500001, 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:37:21.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.96404935699999, 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:37:21.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:37:21.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.613 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.963987011, 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:37:21.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:37:21.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:37:21.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.618 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.963477332, 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:37:21.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.96336626999999, 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.623 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:37:21.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.627 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.96330100399999, 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:37:21.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.96324093800001, 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.632 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:37:21.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.96318424500001, 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:37:21.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:37:21.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.96311491200001, 2024-04-24T00:37:21.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:21.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:37:21.646 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.96302429299999, 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.647 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:37:21.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.96298088, 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:21.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:37:21.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:37:21.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:37:21.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:21.656 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:21.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:21.659 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:21.654+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5910 write 0x1b32e40 thru 0x1b349b3 (0x1b74 bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5912 writesame 0xe5ee77e thru 0xe5f860d data_size 0x56(0x9e90 bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5915 writesame 0xe42d7bd thru 0xe43bf56 data_size 0x31(0xe79a bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5916 read 0x39b8fb7 thru 0x39ba8ea (0x1934 bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5917 writesame 0x67b211c thru 0x67b6b2b data_size 0x3(0x4a10 bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5919 punch from 0xa875ab2 to 0xa879883, (0x3dd1 bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5921 write 0x3532cce thru 0x3534609 (0x193c bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5923 read 0x18a5643 thru 0x18a9a20 (0x43de bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5924 writesame 0xdfed084 thru 0xdff3482 data_size 0x45(0x63ff bytes) 2024-04-24T00:37:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:5926 write 0x1a9225d thru 0x1aa03db (0xe17f bytes) 2024-04-24T00:37:21.665 INFO:teuthology.orchestra.run.smithi064.stdout:5928 writesame 0x77bdeed thru 0x77c0b10 data_size 0x19(0x2c24 bytes) 2024-04-24T00:37:21.677 INFO:teuthology.orchestra.run.smithi064.stdout:5929 write 0x5bf9ed7 thru 0x5c06fa5 (0xd0cf bytes) 2024-04-24T00:37:21.690 INFO:teuthology.orchestra.run.smithi064.stdout:5931 write 0x5fd97a5 thru 0x5fe3366 (0x9bc2 bytes) 2024-04-24T00:37:21.699 INFO:teuthology.orchestra.run.smithi064.stdout:5933 write 0x57e0cff thru 0x57e2ddb (0x20dd bytes) 2024-04-24T00:37:21.703 INFO:teuthology.orchestra.run.smithi064.stdout:5934 writesame 0x8168e0 thru 0x823d86 data_size 0x4d(0xd4a7 bytes) 2024-04-24T00:37:21.708 INFO:teuthology.orchestra.run.smithi064.stdout:5935 read 0x798535e thru 0x798b779 (0x641c bytes) 2024-04-24T00:37:21.724 INFO:teuthology.orchestra.run.smithi064.stdout:5936 trunc from 0xe5f860e to 0x68645e4 2024-04-24T00:37:21.757 INFO:teuthology.orchestra.run.smithi064.stdout:5940 write 0x45bc862 thru 0x45c9c56 (0xd3f5 bytes) 2024-04-24T00:37:21.758 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:21.754+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:21.771 INFO:teuthology.orchestra.run.smithi064.stdout:5943 read 0x33ba0bc thru 0x33c34dd (0x9422 bytes) 2024-04-24T00:37:21.772 INFO:teuthology.orchestra.run.smithi064.stdout:5944 write 0x320c05a thru 0x3217112 (0xb0b9 bytes) 2024-04-24T00:37:21.785 INFO:teuthology.orchestra.run.smithi064.stdout:5945 write 0x4e12db9 thru 0x4e19e28 (0x7070 bytes) 2024-04-24T00:37:21.794 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:37:21.799 INFO:teuthology.orchestra.run.smithi064.stdout:5946 read 0x4a956d2 thru 0x4aa1439 (0xbd68 bytes) 2024-04-24T00:37:21.805 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:21.805 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:21.805 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:21.805 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:21.806 INFO:teuthology.orchestra.run.smithi064.stdout:5948 read 0x66c3779 thru 0x66d28a4 (0xf12c bytes) 2024-04-24T00:37:21.806 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:21.806 INFO:teuthology.orchestra.run.smithi064.stdout:5950 read 0x1a3f6dc thru 0x1a44e09 (0x572e bytes) 2024-04-24T00:37:21.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:21.835 INFO:teuthology.orchestra.run.smithi064.stdout:5952 write 0xcdb1fe9 thru 0xcdb61ee (0x4206 bytes) 2024-04-24T00:37:21.842 INFO:teuthology.orchestra.run.smithi064.stdout:5953 writesame 0x1338802 thru 0x133e37e data_size 0x25(0x5b7d bytes) 2024-04-24T00:37:21.847 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:21.847 INFO:teuthology.orchestra.run.smithi064.stdout:5955 write 0x54cbaec thru 0x54cdfe7 (0x24fc bytes) 2024-04-24T00:37:21.850 INFO:teuthology.orchestra.run.smithi064.stdout:5956 read 0xb23d3f0 thru 0xb24b350 (0xdf61 bytes) 2024-04-24T00:37:21.850 INFO:teuthology.orchestra.run.smithi064.stdout:5957 read 0x54a1d87 thru 0x54abad4 (0x9d4e bytes) 2024-04-24T00:37:21.851 INFO:teuthology.orchestra.run.smithi064.stdout:5958 write 0x8c5d18c thru 0x8c5dfc4 (0xe39 bytes) 2024-04-24T00:37:21.859 INFO:teuthology.orchestra.run.smithi064.stdout:5959 writesame 0x3ec4022 thru 0x3eca4ca data_size 0x1(0x64a9 bytes) 2024-04-24T00:37:21.859 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:21.854+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:21.956 INFO:teuthology.orchestra.run.smithi064.stdout:5960 writesame 0xb33dee thru 0xb3e6fa data_size 0x1(0xa90d bytes) 2024-04-24T00:37:21.961 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:21.958+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:21.962 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:21.962 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:21.962 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:21.962 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:21.962 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:21.979 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:22.069 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:22.067+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:22.103 INFO:teuthology.orchestra.run.smithi064.stdout:5961 write 0x722b64c thru 0x7233dc2 (0x8777 bytes) 2024-04-24T00:37:22.116 INFO:teuthology.orchestra.run.smithi064.stdout:5962 trunc from 0xcdb61ef to 0xd5edb83 2024-04-24T00:37:22.129 INFO:teuthology.orchestra.run.smithi064.stdout:5963 read 0x610c63e thru 0x6118e2a (0xc7ed bytes) 2024-04-24T00:37:22.129 INFO:teuthology.orchestra.run.smithi064.stdout:5964 read 0x121ace0 thru 0x121b86a (0xb8b bytes) 2024-04-24T00:37:22.130 INFO:teuthology.orchestra.run.smithi064.stdout:5966 punch from 0x1b1ed1c to 0x1b2dd3a, (0xf01e bytes) 2024-04-24T00:37:22.130 INFO:teuthology.orchestra.run.smithi064.stdout:5969 read 0x1a1a6ca thru 0x1a1cb9a (0x24d1 bytes) 2024-04-24T00:37:22.132 INFO:teuthology.orchestra.run.smithi064.stdout:5972 writesame 0x3cf4d2b thru 0x3cfcc5b data_size 0x1(0x7f31 bytes) 2024-04-24T00:37:22.157 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:22.157 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:22.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:22.157 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:22.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.157 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110941 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 32768~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.157 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.271442219000001, 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99518982899999997, 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110941 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882159+0000", 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882161+0000", 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882169+0000", 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8480000000000001e-06 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882173+0000", 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6110000000000004e-06 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886136+0000", 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039630810000000002 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886147+0000", 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0484e-05 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876102+0000", 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98995489400000003 2024-04-24T00:37:22.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876434+0000", 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033270799999999997 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.877348+0000", 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091357700000000005 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110942 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 229376~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.271420472999999, 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99714419399999998, 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110942 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5199999999999998e-07 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882182+0000", 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8819999999999999e-06 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882190+0000", 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3629999999999996e-06 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882194+0000", 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5000000000000001e-06 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886176+0000", 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039815329999999998 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886184+0000", 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878386+0000", 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99220202999999996 2024-04-24T00:37:22.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878535+0000", 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001494 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879324+0000", 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078909599999999998 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.167 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110958 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 65536~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.271324161000003, 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.998192368, 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110958 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882277+0000", 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882278+0000", 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6419999999999999e-06 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882284+0000", 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3890000000000004e-06 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882288+0000", 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.138e-06 2024-04-24T00:37:22.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886955+0000", 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046673289999999996 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886964+0000", 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8310000000000007e-06 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879623+0000", 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99265892499999997 2024-04-24T00:37:22.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879818+0000", 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019486800000000001 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880469+0000", 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065093099999999999 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110959 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 262144~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.271264262999999, 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99945926699999998, 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110959 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882337+0000", 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4499999999999997e-07 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882340+0000", 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0259999999999999e-06 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882358+0000", 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8689999999999999e-05 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882365+0000", 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3879999999999997e-06 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886989+0000", 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046244930000000004 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886997+0000", 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880603+0000", 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99360585199999996 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880779+0000", 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175792 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881795+0000", 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001016922 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110974 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 98304~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:22.177 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.271116818000003, 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.000617986, 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110974 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:22.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2100000000000002e-07 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882486+0000", 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.215e-06 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882494+0000", 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9410000000000007e-06 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882500+0000", 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2389999999999999e-06 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888427+0000", 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059265029999999996 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888436+0000", 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2019999999999993e-06 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881936+0000", 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99350027399999996 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.882146+0000", 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020958500000000001 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883102+0000", 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095560600000000001 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110975 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 294912~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:22.182 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.271093336, 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002014129, 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110975 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882508+0000", 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882510+0000", 2024-04-24T00:37:22.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.148e-06 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882515+0000", 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7289999999999997e-06 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882520+0000", 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3290000000000004e-06 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888463+0000", 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059433669999999997 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888470+0000", 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.187e-06 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883245+0000", 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99477520799999997 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883497+0000", 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025132699999999997 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884521+0000", 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001024435 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110990 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 131072~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.270942198, 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0028161980000001, 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110990 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882659+0000", 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4299999999999998e-07 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882661+0000", 2024-04-24T00:37:22.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-06 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882667+0000", 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.879e-06 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882671+0000", 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5349999999999998e-06 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889926+0000", 2024-04-24T00:37:22.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072543970000000001 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889935+0000", 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7070000000000004e-06 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884669+0000", 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99473391700000002 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884841+0000", 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017212600000000001 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885474+0000", 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063308099999999996 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110991 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 327680~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.270923144999998, 2024-04-24T00:37:22.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0041004259999999, 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110991 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882678+0000", 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882684+0000", 2024-04-24T00:37:22.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3219999999999998e-06 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882691+0000", 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882705+0000", 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3813e-05 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889972+0000", 2024-04-24T00:37:22.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072669309999999999 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889981+0000", 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3059999999999998e-06 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885568+0000", 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99558754299999996 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885798+0000", 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023032799999999999 2024-04-24T00:37:22.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886778+0000", 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097934600000000008 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111006 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 163840~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.270787353000003, 2024-04-24T00:37:22.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0048379629999999, 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111006 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882815+0000", 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7549999999999999e-06 2024-04-24T00:37:22.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882822+0000", 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0539999999999997e-06 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882827+0000", 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3619999999999999e-06 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890715+0000", 2024-04-24T00:37:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078878809999999994 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890723+0000", 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6289999999999999e-06 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886854+0000", 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99613063000000002 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887032+0000", 2024-04-24T00:37:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017786200000000001 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887651+0000", 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061942900000000001 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111007 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 360448~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.270767923999998, 2024-04-24T00:37:22.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061255339999999, 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111007 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0400000000000002e-07 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882835+0000", 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7129999999999999e-06 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882842+0000", 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1280000000000002e-06 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882846+0000", 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5600000000000004e-06 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890749+0000", 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079023110000000004 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890758+0000", 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2070000000000001e-06 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887756+0000", 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99699837099999999 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888019+0000", 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263014 2024-04-24T00:37:22.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888958+0000", 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093882600000000003 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111032 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.240834167999999, 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97659819800000003, 2024-04-24T00:37:22.207 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111032 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912767+0000", 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912769+0000", 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6759999999999999e-06 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912777+0000", 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7659999999999998e-06 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912782+0000", 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0590000000000002e-06 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913091+0000", 2024-04-24T00:37:22.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030847099999999999 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913101+0000", 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889077+0000", 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.975975437 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889228+0000", 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015131699999999999 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889364+0000", 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013657900000000001 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111065 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.230038573000002, 2024-04-24T00:37:22.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96634889000000002, 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111065 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923563+0000", 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7100000000000001e-07 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923566+0000", 2024-04-24T00:37:22.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.658e-06 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923577+0000", 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268999999999999e-05 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923588+0000", 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1299e-05 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923668+0000", 2024-04-24T00:37:22.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0012000000000006e-05 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923678+0000", 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9019999999999994e-06 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889478+0000", 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96580058899999999 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889697+0000", 2024-04-24T00:37:22.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218285 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889911+0000", 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000214105 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111116 3.1f 3:f85dfb34:::rbd_data.115e397a9cd4.000000000000000c:head [write 0~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.218837131000001, 2024-04-24T00:37:22.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96849046100000002, 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111116 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934766+0000", 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3580000000000001e-06 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935127+0000", 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036108199999999998 2024-04-24T00:37:22.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935150+0000", 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2756999999999999e-05 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935159+0000", 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935249+0000", 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9470999999999998e-05 2024-04-24T00:37:22.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935263+0000", 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4445e-05 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889988+0000", 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95472527200000001 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890286+0000", 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029795199999999998 2024-04-24T00:37:22.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890670+0000", 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038387799999999999 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902981+0000", 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012310399 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903018+0000", 2024-04-24T00:37:22.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903151+0000", 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013311700000000001 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903254+0000", 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010306699999999999 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111119 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.217857186000003, 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96712422899999995, 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111119 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.224 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935744+0000", 2024-04-24T00:37:22.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.206e-06 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935750+0000", 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.203e-06 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935761+0000", 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1266e-05 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935766+0000", 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2830000000000001e-06 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935841+0000", 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4685999999999996e-05 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935854+0000", 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3135e-05 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902400+0000", 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96654596800000003 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902638+0000", 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023839799999999999 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902868+0000", 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229084 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111159 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.188898301999998, 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.94163226300000002, 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111159 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964704+0000", 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:37:22.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964711+0000", 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4710000000000001e-06 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964727+0000", 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6416e-05 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964736+0000", 2024-04-24T00:37:22.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9209999999999998e-06 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964809+0000", 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3047999999999996e-05 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964821+0000", 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1738e-05 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.905869+0000", 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.941048035 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906136+0000", 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026686400000000001 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906334+0000", 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019838500000000001 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:123125 3.1f 3:facc60b2:::rbd_data.115ee312a896.000000000000001b:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.480537151999997, 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118275349, 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 123125 2024-04-24T00:37:22.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673065+0000", 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673074+0000", 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1500000000000005e-06 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673099+0000", 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692e-05 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673119+0000", 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9683000000000001e-05 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673252+0000", 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133354 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673490+0000", 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023785700000000001 2024-04-24T00:37:22.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.680775+0000", 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007284858 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791134+0000", 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110358923 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791167+0000", 2024-04-24T00:37:22.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2985999999999999e-05 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791260+0000", 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3060999999999996e-05 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791339+0000", 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8844999999999995e-05 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:123125 3.1f e168/163 ondisk, result = 0)", 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.454651396999999, 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092199987999999997, 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698951+0000", 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7460000000000001e-06 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698959+0000", 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.746e-06 2024-04-24T00:37:22.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698988+0000", 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9257000000000001e-05 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.699001+0000", 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3362e-05 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791078+0000", 2024-04-24T00:37:22.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092076651999999995 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791120+0000", 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1789000000000003e-05 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791149+0000", 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9436000000000001e-05 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143625 3.18 3:1b6251e3:::rbd_data.115e8dbf22aa.0000000000000039:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.498956229000001, 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033012196000000001, 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143625 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654647+0000", 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654660+0000", 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.309e-05 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654691+0000", 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0709e-05 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654717+0000", 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6267000000000001e-05 2024-04-24T00:37:22.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654859+0000", 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014176400000000001 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.655117+0000", 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025819200000000002 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.656922+0000", 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00180528 2024-04-24T00:37:22.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687423+0000", 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030501117000000001 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687502+0000", 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9127000000000003e-05 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687584+0000", 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2139999999999996e-05 2024-04-24T00:37:22.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687656+0000", 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1989999999999993e-05 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143678 3.b 3:d70c067f:::rbd_data.115e8dbf22aa.000000000000005e:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.316668479000001, 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18713334100000001, 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143678 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836936+0000", 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9829999999999998e-06 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836953+0000", 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7016000000000001e-05 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836986+0000", 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3319999999999999e-05 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837001+0000", 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837183+0000", 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018131700000000001 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837430+0000", 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024749599999999999 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.847946+0000", 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01051566 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023623+0000", 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17567688400000001 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023871+0000", 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024814800000000001 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023984+0000", 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011326499999999999 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024065+0000", 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1129000000000002e-05 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:143678 3.b e183/160 ondisk, result = 0)", 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.271262178999997, 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14174242100000001, 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882340+0000", 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882350+0000", 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882384+0000", 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3923e-05 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882407+0000", 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2619000000000001e-05 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023712+0000", 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.141305088 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023728+0000", 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.607e-05 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024081+0000", 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352648 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.259 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:22.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:22.261 INFO:teuthology.orchestra.run.smithi064.stdout:5973 read 0x8396bbc thru 0x8398081 (0x14c6 bytes) 2024-04-24T00:37:22.261 INFO:teuthology.orchestra.run.smithi064.stdout:5975 write 0x8ba7c51 thru 0x8bb40a5 (0xc455 bytes) 2024-04-24T00:37:22.261 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:22.255+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:22.265 INFO:teuthology.orchestra.run.smithi064.stdout:5977 read 0x7d3516c thru 0x7d36010 (0xea5 bytes) 2024-04-24T00:37:22.265 INFO:teuthology.orchestra.run.smithi064.stdout:5978 read 0x9f9f866 thru 0x9facdd3 (0xd56e bytes) 2024-04-24T00:37:22.265 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:22.284 INFO:teuthology.orchestra.run.smithi064.stdout:5980 trunc from 0xd5edb83 to 0x417751e 2024-04-24T00:37:22.346 INFO:teuthology.orchestra.run.smithi064.stdout:5981 write 0xdccd97c thru 0xdcdb097 (0xd71c bytes) 2024-04-24T00:37:22.356 INFO:teuthology.orchestra.run.smithi064.stdout:5982 writesame 0x1c731a2 thru 0x1c7ea64 data_size 0x1d(0xb8c3 bytes) 2024-04-24T00:37:22.361 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:22.358+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:22.364 INFO:teuthology.orchestra.run.smithi064.stdout:5984 write 0xc3cce45 thru 0xc3d2496 (0x5652 bytes) 2024-04-24T00:37:22.375 INFO:teuthology.orchestra.run.smithi064.stdout:5985 write 0x97973ea thru 0x9797e7b (0xa92 bytes) 2024-04-24T00:37:22.389 INFO:teuthology.orchestra.run.smithi064.stdout:5986 trunc from 0xdcdb098 to 0x9ce9433 2024-04-24T00:37:22.420 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:22.420 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:22.420 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:22.420 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:22.430 INFO:teuthology.orchestra.run.smithi064.stdout:5987 write 0xd0bc1b7 thru 0xd0c44ff (0x8349 bytes) 2024-04-24T00:37:22.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:22.440 INFO:teuthology.orchestra.run.smithi064.stdout:5988 read 0x841b7c1 thru 0x84207eb (0x502b bytes) 2024-04-24T00:37:22.440 INFO:teuthology.orchestra.run.smithi064.stdout:5989 write 0x223db9e thru 0x2246d82 (0x91e5 bytes) 2024-04-24T00:37:22.449 INFO:teuthology.orchestra.run.smithi064.stdout:5990 trunc from 0xd0c4500 to 0xe2b6f92 2024-04-24T00:37:22.460 INFO:teuthology.orchestra.run.smithi064.stdout:5991 read 0x4356d00 thru 0x435b1bc (0x44bd bytes) 2024-04-24T00:37:22.461 INFO:teuthology.orchestra.run.smithi064.stdout:5992 write 0x4a57479 thru 0x4a58aca (0x1652 bytes) 2024-04-24T00:37:22.461 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:22.458+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:22.469 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:22.469 INFO:teuthology.orchestra.run.smithi064.stdout:5995 punch from 0x42fae80 to 0x4309026, (0xe1a6 bytes) 2024-04-24T00:37:22.469 INFO:teuthology.orchestra.run.smithi064.stdout:5997 punch from 0xb2f905e to 0xb308127, (0xf0c9 bytes) 2024-04-24T00:37:22.469 INFO:teuthology.orchestra.run.smithi064.stdout:5998 read 0xdebd42 thru 0xdf58ed (0x9bac bytes) 2024-04-24T00:37:22.470 INFO:teuthology.orchestra.run.smithi064.stdout:compare and write disabled 2024-04-24T00:37:22.498 INFO:teuthology.orchestra.run.smithi064.stdout:checking clone #40, image image_client.0-clone40 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent41 2024-04-24T00:37:22.563 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:22.558+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:22.572 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:22.572 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:22.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:22.664 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:22.658+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:22.765 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:22.759+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.32959492800001, 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:37:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:37:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:37:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:37:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 175.27408344200001, 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:37:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:37:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:37:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.17350679399999, 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:37:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.13194184, 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:37:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.568210278, 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 115.065236881, 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:37:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:37:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.82692264799999, 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791982225, 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:37:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:37:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:37:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791961352, 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791937973, 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:37:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791916431, 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:37:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:37:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791894911, 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791848754, 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791824635, 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:22.832 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.79180123099999, 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:37:22.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:37:22.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.79177345799999, 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:37:22.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:37:22.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791752045, 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.841 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:37:22.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.791731454, 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:37:22.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.575661212, 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:37:22.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:37:22.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:37:22.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:37:22.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:37:22.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:37:22.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:37:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:37:22.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.547690308, 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:37:22.856 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:22.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:22.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:37:22.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:37:22.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.860 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:22.860 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:22.860 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:22.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:22.906 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:22.903+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:22.967 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:22.962+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:23.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:23.050 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:23.050 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:23.051 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:23.051 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:23.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:23.113 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:23.111+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:23.165 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:23.165 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:23.168 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:23.163+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:23.182 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:23.182 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:23.199 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:23.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:23.217 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:23.233 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:23.233 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:23.238 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:23.239 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:23.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:23.260 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:23.260 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:23.281 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:23.281 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:23.282 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:23.279+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:23.302 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:23.302 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:23.370 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:23.367+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:23.376 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:23.376 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:23.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:23.477 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:23.475+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:23.559 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:23.559 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:23.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:23.574 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:23.567+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:23.674 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:23.671+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:23.746 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:23.746 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:23.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:23.801 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:23.799+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:23.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:37:23.878 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:23.874+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:23.913 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:23.914 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:23.914 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-24T00:37:23.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-24T00:37:23.933 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:23.934 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:23.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:24.000 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:23.998+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:24.078 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-04-24T00:37:24.078 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-04-24T00:37:24.078 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-04-24T00:37:24.078 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-04-24T00:37:24.078 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-04-24T00:37:24.078 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-04-24T00:37:24.079 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-04-24T00:37:24.079 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-04-24T00:37:24.079 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-04-24T00:37:24.079 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-04-24T00:37:24.079 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-04-24T00:37:24.079 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-04-24T00:37:24.080 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:24.075+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T00:37:24.081 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T00:37:24.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-24T00:37:24.089 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:24.090 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:24.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:24.181 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:24.175+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:24.267 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:24.267 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:24.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:24.282 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:24.278+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-04-24T00:37:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-04-24T00:37:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-04-24T00:37:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-04-24T00:37:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-04-24T00:37:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-04-24T00:37:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-04-24T00:37:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-04-24T00:37:24.284 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-04-24T00:37:24.284 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-04-24T00:37:24.284 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-04-24T00:37:24.284 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-04-24T00:37:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T00:37:24.285 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T00:37:24.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-24T00:37:24.383 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:24.378+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:24.443 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:24.443 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:24.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:24.495 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-04-24T00:37:24.495 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-04-24T00:37:24.495 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-04-24T00:37:24.495 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-04-24T00:37:24.495 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-04-24T00:37:24.495 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-04-24T00:37:24.495 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-04-24T00:37:24.496 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-04-24T00:37:24.496 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-04-24T00:37:24.496 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-04-24T00:37:24.496 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-04-24T00:37:24.496 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-04-24T00:37:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T00:37:24.498 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T00:37:24.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-04-24T00:37:24.501 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:24.498+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:24.585 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:24.583+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:24.624 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:24.624 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:24.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:24.655 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-04-24T00:37:24.655 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-04-24T00:37:24.655 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-04-24T00:37:24.655 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-04-24T00:37:24.655 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-04-24T00:37:24.655 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-04-24T00:37:24.655 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-04-24T00:37:24.656 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-04-24T00:37:24.656 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-04-24T00:37:24.656 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-04-24T00:37:24.656 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-04-24T00:37:24.656 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-04-24T00:37:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T00:37:24.658 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T00:37:24.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-04-24T00:37:24.702 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:24.699+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:24.759 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:24.759 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:24.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:24.820 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:37:24.821 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:24.818+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-04-24T00:37:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-04-24T00:37:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T00:37:24.874 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T00:37:24.874 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-04-24T00:37:24.910 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:24.910 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:24.921 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:24.919+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:24.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:25.013 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:25.010+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:25.090 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:25.086+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:25.092 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-04-24T00:37:25.092 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-04-24T00:37:25.093 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-04-24T00:37:25.094 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:25.094 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T00:37:25.097 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-24T00:37:25.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:25.192 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:25.187+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:25.276 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:25.276 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:25.291 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:25.292 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:25.287+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:25.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:25.393 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:25.391+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:25.413 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:25.414 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:25.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:25.481 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:25.481 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:25.493 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:25.490+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:25.497 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:25.498 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:25.514 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:25.515 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:25.531 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:25.531 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:25.548 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:25.548 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:25.550 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:25.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:25.572 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:25.572 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:25.593 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:25.593 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:25.605 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:25.602+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:25.612 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:25.612 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:25.695 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:25.691+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:25.723 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:25.723 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:25.723 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:25.724 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:25.724 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:25.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:25.797 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:25.794+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 224.93892845100001, 2024-04-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.873 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 192.00696276299999, 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:37:25.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:37:25.879 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 192.005714015, 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:37:25.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:25.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 182.84808991099999, 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:37:25.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:37:25.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:37:25.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 182.83459536800001, 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:37:25.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.57281425299999, 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.896 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:37:25.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:37:25.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:37:25.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.31922811000001, 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.901 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:37:25.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:37:25.906 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.291554246, 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.907 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 164.79753142600001, 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:37:25.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:37:25.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:25.917 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 164.78852259199999, 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:37:25.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:37:25.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.28682364400001, 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.922 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:37:25.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:37:25.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:37:25.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:37:25.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:37:25.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.28676129799999, 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:37:25.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.28625161900001, 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.932 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.936 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.28614055700001, 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.937 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:37:25.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:37:25.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.286075291, 2024-04-24T00:37:25.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:25.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.945 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.286015225, 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.946 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:37:25.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.285958532, 2024-04-24T00:37:25.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.285889199, 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:37:25.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.28579858000001, 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.28575516699999, 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:25.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:25.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:25.966 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.967 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.968 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:25.968 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:25.968 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:25.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:25.970 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:25.966+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:26.070 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:26.066+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:26.118 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:26.118 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:26.118 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:26.118 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:26.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:26.181 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:26.179+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:26.254 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:26.254 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:26.255 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:26.255 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:26.255 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:26.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:26.273 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:26.270+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:26.374 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:26.370+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:26.438 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110941 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 32768~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551695877, 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99518982899999997, 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110941 2024-04-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882159+0000", 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882161+0000", 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882169+0000", 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8480000000000001e-06 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882173+0000", 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6110000000000004e-06 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886136+0000", 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039630810000000002 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886147+0000", 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0484e-05 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876102+0000", 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98995489400000003 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876434+0000", 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033270799999999997 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.877348+0000", 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091357700000000005 2024-04-24T00:37:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110942 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 229376~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551674130999999, 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99714419399999998, 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110942 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5199999999999998e-07 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882182+0000", 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8819999999999999e-06 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882190+0000", 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3629999999999996e-06 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882194+0000", 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5000000000000001e-06 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886176+0000", 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039815329999999998 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886184+0000", 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-04-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878386+0000", 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99220202999999996 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878535+0000", 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001494 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879324+0000", 2024-04-24T00:37:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078909599999999998 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110958 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 65536~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551577819000002, 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.998192368, 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110958 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882277+0000", 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882278+0000", 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6419999999999999e-06 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882284+0000", 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3890000000000004e-06 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882288+0000", 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.138e-06 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886955+0000", 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046673289999999996 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886964+0000", 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8310000000000007e-06 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879623+0000", 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99265892499999997 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879818+0000", 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019486800000000001 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880469+0000", 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065093099999999999 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110959 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 262144~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551517920999999, 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99945926699999998, 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110959 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882337+0000", 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4499999999999997e-07 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882340+0000", 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0259999999999999e-06 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882358+0000", 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8689999999999999e-05 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882365+0000", 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3879999999999997e-06 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886989+0000", 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046244930000000004 2024-04-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886997+0000", 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880603+0000", 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99360585199999996 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880779+0000", 2024-04-24T00:37:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175792 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881795+0000", 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001016922 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110974 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 98304~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551370476000002, 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.000617986, 2024-04-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110974 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2100000000000002e-07 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882486+0000", 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.215e-06 2024-04-24T00:37:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882494+0000", 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9410000000000007e-06 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882500+0000", 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2389999999999999e-06 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888427+0000", 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059265029999999996 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888436+0000", 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2019999999999993e-06 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881936+0000", 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99350027399999996 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.882146+0000", 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020958500000000001 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883102+0000", 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095560600000000001 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110975 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 294912~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551346993999999, 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002014129, 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110975 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882508+0000", 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882510+0000", 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.148e-06 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882515+0000", 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7289999999999997e-06 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882520+0000", 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3290000000000004e-06 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888463+0000", 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059433669999999997 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888470+0000", 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.187e-06 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883245+0000", 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99477520799999997 2024-04-24T00:37:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883497+0000", 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025132699999999997 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884521+0000", 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001024435 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110990 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 131072~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551195856, 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0028161980000001, 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110990 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882659+0000", 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4299999999999998e-07 2024-04-24T00:37:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882661+0000", 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-06 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882667+0000", 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.879e-06 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882671+0000", 2024-04-24T00:37:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5349999999999998e-06 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889926+0000", 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072543970000000001 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889935+0000", 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7070000000000004e-06 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884669+0000", 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99473391700000002 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884841+0000", 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017212600000000001 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885474+0000", 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063308099999999996 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110991 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 327680~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551176802999997, 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0041004259999999, 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110991 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882678+0000", 2024-04-24T00:37:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882684+0000", 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3219999999999998e-06 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882691+0000", 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882705+0000", 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3813e-05 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889972+0000", 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072669309999999999 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889981+0000", 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3059999999999998e-06 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885568+0000", 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99558754299999996 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885798+0000", 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023032799999999999 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886778+0000", 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097934600000000008 2024-04-24T00:37:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111006 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 163840~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551041011000002, 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0048379629999999, 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111006 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882815+0000", 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7549999999999999e-06 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882822+0000", 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0539999999999997e-06 2024-04-24T00:37:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882827+0000", 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3619999999999999e-06 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890715+0000", 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078878809999999994 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890723+0000", 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6289999999999999e-06 2024-04-24T00:37:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886854+0000", 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99613063000000002 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887032+0000", 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017786200000000001 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887651+0000", 2024-04-24T00:37:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061942900000000001 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111007 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 360448~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.551021581999997, 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061255339999999, 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111007 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0400000000000002e-07 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882835+0000", 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7129999999999999e-06 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882842+0000", 2024-04-24T00:37:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1280000000000002e-06 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882846+0000", 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5600000000000004e-06 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890749+0000", 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079023110000000004 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890758+0000", 2024-04-24T00:37:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2070000000000001e-06 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887756+0000", 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99699837099999999 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888019+0000", 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263014 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888958+0000", 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093882600000000003 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111032 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.521087825999999, 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97659819800000003, 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111032 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912767+0000", 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912769+0000", 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6759999999999999e-06 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912777+0000", 2024-04-24T00:37:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7659999999999998e-06 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912782+0000", 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0590000000000002e-06 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913091+0000", 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030847099999999999 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913101+0000", 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889077+0000", 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.975975437 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889228+0000", 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015131699999999999 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889364+0000", 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013657900000000001 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111065 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.510292231000001, 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96634889000000002, 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111065 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923563+0000", 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7100000000000001e-07 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923566+0000", 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.658e-06 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923577+0000", 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268999999999999e-05 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923588+0000", 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1299e-05 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923668+0000", 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0012000000000006e-05 2024-04-24T00:37:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923678+0000", 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9019999999999994e-06 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889478+0000", 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96580058899999999 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889697+0000", 2024-04-24T00:37:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218285 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889911+0000", 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000214105 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111116 3.1f 3:f85dfb34:::rbd_data.115e397a9cd4.000000000000000c:head [write 0~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.499090789, 2024-04-24T00:37:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96849046100000002, 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111116 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934766+0000", 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3580000000000001e-06 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935127+0000", 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036108199999999998 2024-04-24T00:37:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935150+0000", 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2756999999999999e-05 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935159+0000", 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935249+0000", 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9470999999999998e-05 2024-04-24T00:37:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935263+0000", 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4445e-05 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889988+0000", 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95472527200000001 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890286+0000", 2024-04-24T00:37:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029795199999999998 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890670+0000", 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038387799999999999 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902981+0000", 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012310399 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903018+0000", 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903151+0000", 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013311700000000001 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903254+0000", 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010306699999999999 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111119 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.498110844000003, 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96712422899999995, 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111119 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935744+0000", 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.206e-06 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935750+0000", 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.203e-06 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935761+0000", 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1266e-05 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935766+0000", 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2830000000000001e-06 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935841+0000", 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4685999999999996e-05 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935854+0000", 2024-04-24T00:37:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3135e-05 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902400+0000", 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96654596800000003 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902638+0000", 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023839799999999999 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902868+0000", 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229084 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111159 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.469151959999998, 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.94163226300000002, 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111159 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964704+0000", 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964711+0000", 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4710000000000001e-06 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964727+0000", 2024-04-24T00:37:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6416e-05 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964736+0000", 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9209999999999998e-06 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964809+0000", 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3047999999999996e-05 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964821+0000", 2024-04-24T00:37:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1738e-05 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.905869+0000", 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.941048035 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906136+0000", 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026686400000000001 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906334+0000", 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019838500000000001 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:123125 3.1f 3:facc60b2:::rbd_data.115ee312a896.000000000000001b:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.760790810000003, 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118275349, 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 123125 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673065+0000", 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673074+0000", 2024-04-24T00:37:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1500000000000005e-06 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673099+0000", 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692e-05 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673119+0000", 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9683000000000001e-05 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673252+0000", 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133354 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673490+0000", 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023785700000000001 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.680775+0000", 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007284858 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791134+0000", 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110358923 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791167+0000", 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2985999999999999e-05 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791260+0000", 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3060999999999996e-05 2024-04-24T00:37:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791339+0000", 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8844999999999995e-05 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:123125 3.1f e168/163 ondisk, result = 0)", 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.734905054999999, 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092199987999999997, 2024-04-24T00:37:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698951+0000", 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7460000000000001e-06 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698959+0000", 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.746e-06 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698988+0000", 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9257000000000001e-05 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.699001+0000", 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3362e-05 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791078+0000", 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092076651999999995 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791120+0000", 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1789000000000003e-05 2024-04-24T00:37:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791149+0000", 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9436000000000001e-05 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143625 3.18 3:1b6251e3:::rbd_data.115e8dbf22aa.0000000000000039:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.779209887, 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033012196000000001, 2024-04-24T00:37:26.527 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143625 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654647+0000", 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654660+0000", 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.309e-05 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654691+0000", 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0709e-05 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654717+0000", 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6267000000000001e-05 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654859+0000", 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014176400000000001 2024-04-24T00:37:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.655117+0000", 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025819200000000002 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.656922+0000", 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00180528 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687423+0000", 2024-04-24T00:37:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030501117000000001 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687502+0000", 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9127000000000003e-05 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687584+0000", 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2139999999999996e-05 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687656+0000", 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1989999999999993e-05 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143678 3.b 3:d70c067f:::rbd_data.115e8dbf22aa.000000000000005e:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.596922137, 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18713334100000001, 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143678 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836936+0000", 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9829999999999998e-06 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836953+0000", 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7016000000000001e-05 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836986+0000", 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3319999999999999e-05 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837001+0000", 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837183+0000", 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018131700000000001 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837430+0000", 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024749599999999999 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.847946+0000", 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01051566 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023623+0000", 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17567688400000001 2024-04-24T00:37:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023871+0000", 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024814800000000001 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023984+0000", 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011326499999999999 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024065+0000", 2024-04-24T00:37:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1129000000000002e-05 2024-04-24T00:37:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.541 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:143678 3.b e183/160 ondisk, result = 0)", 2024-04-24T00:37:26.541 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:26.541 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.551515836999997, 2024-04-24T00:37:26.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14174242100000001, 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882340+0000", 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882350+0000", 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882384+0000", 2024-04-24T00:37:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3923e-05 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882407+0000", 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2619000000000001e-05 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023712+0000", 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.141305088 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023728+0000", 2024-04-24T00:37:26.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.607e-05 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024081+0000", 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352648 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:26.545 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:26.547 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:26.543+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:26.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:26.648 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:26.642+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:26.718 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:26.719 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:26.719 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:26.719 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:26.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:26.748 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:26.746+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:26.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:37:26.849 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:26.843+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:26.860 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:26.860 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:26.860 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:26.860 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:26.860 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:26.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:26.950 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:26.946+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:27.050 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:27.047+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:27.057 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 179.62105163199999, 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:37:27.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:27.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:27.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 179.56554014599999, 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:37:27.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:27.067 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.464963498, 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:37:27.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.42339854400001, 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:37:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:37:27.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.85966698199999, 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:37:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 119.356693585, 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:37:27.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:37:27.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.11837935200001, 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:27.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.092 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.083438929, 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.083418056, 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.097 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.083394677, 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.105 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.083373135, 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.106 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:37:27.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:37:27.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.083351615, 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.114 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.083305458, 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:37:27.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.083281339, 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:37:27.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:37:27.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.123 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.08325793500001, 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.08323016200001, 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.08320874899999, 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.136 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.083188158, 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:37:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.867117916, 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:37:27.141 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:27.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:37:27.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.839147012, 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:37:27.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:37:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:37:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.151 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:27.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:27.153 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:27.150+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:27.253 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:27.251+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:27.314 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:27.314 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:27.314 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:27.314 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:27.328 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:27.369 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:27.367+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:27.456 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:27.451+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:27.498 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:27.498 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:27.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:27.556 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:27.554+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:27.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:27.657 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:27.654+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:27.685 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 67.441047830000002, 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:37:27.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:27.692 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 67.411280348999995, 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:37:27.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:37:27.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126791 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 112640~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.041715320000002, 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074437884999999995, 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.696 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126791 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640034+0000", 2024-04-24T00:37:27.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640044+0000", 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2660000000000007e-06 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640050+0000", 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713112+0000", 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073062031999999999 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713345+0000", 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000232616 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714469+0000", 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011237669999999999 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.700 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126792 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 116736~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.041686546999998, 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075734135999999994, 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126792 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3399999999999999e-07 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640063+0000", 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7e-06 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640072+0000", 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0729999999999994e-06 2024-04-24T00:37:27.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640084+0000", 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2486e-05 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714588+0000", 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074503102000000002 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714763+0000", 2024-04-24T00:37:27.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175885 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715794+0000", 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001030256 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126793 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 120832~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.041652757000001, 2024-04-24T00:37:27.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076915124000000001, 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126793 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640094+0000", 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640097+0000", 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9799999999999998e-06 2024-04-24T00:37:27.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640107+0000", 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6229999999999992e-06 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640123+0000", 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5892999999999998e-05 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715884+0000", 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075761384000000001 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.716061+0000", 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176727 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717009+0000", 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094787400000000003 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126794 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 124928~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.041614078000002, 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078057396000000001, 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126794 2024-04-24T00:37:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640133+0000", 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-07 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640136+0000", 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1e-06 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640145+0000", 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.516e-06 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640152+0000", 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717088+0000", 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076936302999999998 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717201+0000", 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112704 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718189+0000", 2024-04-24T00:37:27.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098839299999999995 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126795 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 129024~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.041585191999999, 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079207986999999994, 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126795 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640162+0000", 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7400000000000003e-07 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640165+0000", 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1860000000000001e-06 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640174+0000", 2024-04-24T00:37:27.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5429999999999999e-06 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640181+0000", 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8029999999999999e-06 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718272+0000", 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078091277000000001 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718443+0000", 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171027 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.719369+0000", 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092547699999999996 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127300 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 229376~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:27.719 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.019328932999997, 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073789773000000003, 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127300 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662418+0000", 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.203e-06 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662439+0000", 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0713e-05 2024-04-24T00:37:27.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662460+0000", 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.105e-05 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662488+0000", 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7966e-05 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735368+0000", 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072880167999999995 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735555+0000", 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018692399999999999 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736207+0000", 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065174899999999997 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127301 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 233472~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.019227974000003, 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074400210999999994, 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127301 2024-04-24T00:37:27.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662520+0000", 2024-04-24T00:37:27.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8700000000000001e-06 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662527+0000", 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7830000000000001e-06 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662549+0000", 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2379e-05 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662564+0000", 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.473e-05 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736281+0000", 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073717486999999998 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736386+0000", 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104594 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736918+0000", 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053236799999999997 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127302 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.019160049, 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075297446000000004, 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127302 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662587+0000", 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096e-06 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662608+0000", 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662624+0000", 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5563e-05 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662639+0000", 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4766999999999999e-05 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737013+0000", 2024-04-24T00:37:27.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074374061000000005 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737195+0000", 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182201 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737884+0000", 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068866800000000003 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127303 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.019086170999998, 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075989524000000003, 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127303 2024-04-24T00:37:27.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662662+0000", 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5400000000000001e-06 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662668+0000", 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1970000000000001e-06 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662689+0000", 2024-04-24T00:37:27.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0953000000000001e-05 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662703+0000", 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737962+0000", 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075258932000000001 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738074+0000", 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011173800000000001 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738649+0000", 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057545399999999998 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127304 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:27.737 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.019021838, 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076894431999999999, 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127304 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662725+0000", 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0359999999999999e-06 2024-04-24T00:37:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662732+0000", 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6440000000000003e-06 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662751+0000", 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8583000000000001e-05 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662768+0000", 2024-04-24T00:37:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7336000000000001e-05 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738744+0000", 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075976005999999999 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738924+0000", 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017996900000000001 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739619+0000", 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069485800000000002 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127305 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.018957563000001, 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077646889999999996, 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.742 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127305 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662790+0000", 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662796+0000", 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3659999999999997e-06 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662819+0000", 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2503000000000001e-05 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662834+0000", 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739755+0000", 2024-04-24T00:37:27.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076920905999999997 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739876+0000", 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121445 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740435+0000", 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055911299999999997 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127306 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.018889999999999, 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078485103000000001, 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127306 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.747 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662857+0000", 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1599999999999999e-06 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662864+0000", 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3829999999999997e-06 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662883+0000", 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9476000000000001e-05 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662897+0000", 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3485999999999999e-05 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740564+0000", 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077667768999999998 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740736+0000", 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017116700000000001 2024-04-24T00:37:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741341+0000", 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060566199999999998 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127307 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.018830594999997, 2024-04-24T00:37:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079342509000000006, 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127307 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:27.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662917+0000", 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662922+0000", 2024-04-24T00:37:27.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662939+0000", 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833e-05 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662954+0000", 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741518+0000", 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078564118000000002 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741709+0000", 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191337 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.742258+0000", 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054904499999999996 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127559 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 243712~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.998046289999998, 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073184308000000003, 2024-04-24T00:37:27.756 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127559 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683701+0000", 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2300000000000001e-06 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683724+0000", 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.321e-05 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683745+0000", 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.101e-05 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683761+0000", 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756124+0000", 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072363532999999994 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756327+0000", 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020239999999999999 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756884+0000", 2024-04-24T00:37:27.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055760300000000005 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127560 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 247808~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.997962551000001, 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073924043999999994, 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.761 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127560 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:27.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683785+0000", 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1829999999999999e-06 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683792+0000", 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2799999999999998e-06 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683810+0000", 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8076000000000001e-05 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683842+0000", 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756969+0000", 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073126925999999995 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757172+0000", 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020269900000000001 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757708+0000", 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053609999999999997 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127561 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 251904~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.997881223, 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074620636000000004, 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127561 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:27.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683866+0000", 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.127e-06 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683873+0000", 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683893+0000", 2024-04-24T00:37:27.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9300000000000002e-05 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683954+0000", 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1148999999999993e-05 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757783+0000", 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073829025000000006 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757939+0000", 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582700000000001 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758486+0000", 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054703200000000003 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127562 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 256000~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:27.772 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.997759627999997, 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075112683, 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127562 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:27.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683988+0000", 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.443e-06 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684000+0000", 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.252e-05 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684025+0000", 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4956e-05 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684044+0000", 2024-04-24T00:37:27.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8770000000000002e-05 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758566+0000", 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074522094999999997 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758669+0000", 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010247399999999999 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759099+0000", 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043042500000000001 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127563 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 260096~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.997665796, 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075791184999999997, 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:27.777 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127563 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:27.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684082+0000", 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684089+0000", 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4889999999999998e-06 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684115+0000", 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5643000000000002e-05 2024-04-24T00:37:27.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684133+0000", 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7842999999999998e-05 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759174+0000", 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075041103999999997 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759329+0000", 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015504200000000001 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:27.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759872+0000", 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054272399999999996 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:27.781 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:27.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:27.784 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:27.778+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:27.788 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:27.789 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:27.806 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:27.806 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:27.822 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:27.823 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:27.842 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:27.842 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:27.845 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:37:27.869 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:27.869 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:27.892 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:27.892 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:27.887+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:27.925 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:27.926 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:27.929 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:27.929 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:27.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:27.952 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:27.953 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:27.987 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:27.982+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:28.085 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:28.085 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:28.086 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:28.083+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:28.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:28.186 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:28.183+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:28.253 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:28.253 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:28.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:28.309 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:28.307+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:28.388 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:28.383+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:28.399 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:28.399 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:28.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:28.489 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:28.486+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:28.537 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:28.537 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:28.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:28.593 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:28.590+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:28.691 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:28.687+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:28.719 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:28.720 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:28.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:28.792 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:28.786+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:28.863 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:28.863 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:28.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:28.892 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:28.887+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:28.994 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:28.991+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:29.073 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:29.074 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:29.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:29.133 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:29.130+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:29.196 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:29.191+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:29.206 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:29.206 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:29.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:29.297 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:29.295+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:29.385 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:29.385 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:29.398 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:29.395+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:29.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:29.499 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:29.495+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:29.522 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:29.522 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:29.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:29.600 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:29.594+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:29.670 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:29.670 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:29.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:29.729 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:29.726+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:29.802 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:29.799+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:29.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:37:29.870 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:29.870 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:29.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:29.913 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:29.911+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:29.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:30.003 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:29.998+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:30.006 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:30.007 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:30.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:30.098 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:30.098 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:30.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:30.119 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:30.119 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:30.115+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:30.136 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:30.153 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:30.170 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:30.170 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:30.185 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:30.185 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:30.190 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:30.191 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:30.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:30.205 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:30.202+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:30.209 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:30.209 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:30.230 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:30.230 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:30.264 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:30.265 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:30.320 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:30.320 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:30.320 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:30.320 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:30.320 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:30.315+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:30.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:30.406 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:30.402+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:30.421 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:37:30.422 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:37:30.442 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-24T00:37:29.706511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1637501794,"num_objects":1250,"num_object_clones":0,"num_object_copies":2500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":177730,"num_read_kb":1538756,"num_write":28877,"num_write_kb":1020091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3261,"num_bytes_recovered":284359903,"num_keys_recovered":395,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20230,"ondisk_log_size":20230,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":416896,"kb_used_data":84900,"kb_used_omap":9,"kb_used_meta":331382,"kb_avail":565814144,"statfs":{"total":579820584960,"available":579393683456,"internally_reserved":0,"allocated":86937600,"data_stored":742942518,"data_compressed":14777803,"data_compressed_allocated":62545920,"data_compressed_original":730682510,"omap_allocated":9567,"internal_metadata":339335841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-233937727,"num_objects":-225,"num_object_clones":0,"num_object_copies":-450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-225,"num_whiteouts":0,"num_read":1138,"num_read_kb":976,"num_write":400,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.882058"},"pg_stats":[{"pgid":"3.1f","version":"215'801","reported_seq":5543,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.848886+0000","last_change":"2024-04-24T00:37:27.848886+0000","last_active":"2024-04-24T00:37:27.848886+0000","last_peered":"2024-04-24T00:37:27.848886+0000","last_clean":"2024-04-24T00:37:27.848886+0000","last_became_active":"2024-04-24T00:36:56.589008+0000","last_became_peered":"2024-04-24T00:36:56.589008+0000","last_unstale":"2024-04-24T00:37:27.848886+0000","last_undegraded":"2024-04-24T00:37:27.848886+0000","last_fullsized":"2024-04-24T00:37:27.848886+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:45:50.789370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028721390000000001,"stat_sum":{"num_bytes":68574267,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":6111,"num_read_kb":55827,"num_write":1276,"num_write_kb":38946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":14807962,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.1e","version":"215'917","reported_seq":4965,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.864160+0000","last_change":"2024-04-24T00:37:27.864160+0000","last_active":"2024-04-24T00:37:27.864160+0000","last_peered":"2024-04-24T00:37:27.864160+0000","last_clean":"2024-04-24T00:37:27.864160+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:37:27.864160+0000","last_undegraded":"2024-04-24T00:37:27.864160+0000","last_fullsized":"2024-04-24T00:37:27.864160+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:52:35.971456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032961850000000001,"stat_sum":{"num_bytes":34029517,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":7227,"num_read_kb":34596,"num_write":1220,"num_write_kb":26848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.1d","version":"215'518","reported_seq":6575,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.851173+0000","last_change":"2024-04-24T00:37:27.851173+0000","last_active":"2024-04-24T00:37:27.851173+0000","last_peered":"2024-04-24T00:37:27.851173+0000","last_clean":"2024-04-24T00:37:27.851173+0000","last_became_active":"2024-04-24T00:34:28.726196+0000","last_became_peered":"2024-04-24T00:34:28.726196+0000","last_unstale":"2024-04-24T00:37:27.851173+0000","last_undegraded":"2024-04-24T00:37:27.851173+0000","last_fullsized":"2024-04-24T00:37:27.851173+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:24.422290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043968230000000002,"stat_sum":{"num_bytes":94116903,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":7101,"num_read_kb":66678,"num_write":757,"num_write_kb":27391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":458933,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.1c","version":"215'659","reported_seq":5167,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.993613+0000","last_change":"2024-04-24T00:37:27.847196+0000","last_active":"2024-04-24T00:37:27.993613+0000","last_peered":"2024-04-24T00:37:27.993613+0000","last_clean":"2024-04-24T00:37:27.993613+0000","last_became_active":"2024-04-24T00:34:31.747052+0000","last_became_peered":"2024-04-24T00:34:31.747052+0000","last_unstale":"2024-04-24T00:37:27.993613+0000","last_undegraded":"2024-04-24T00:37:27.993613+0000","last_fullsized":"2024-04-24T00:37:27.993613+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:54:32.915281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021459109999999999,"stat_sum":{"num_bytes":57418864,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":4846,"num_read_kb":60987,"num_write":841,"num_write_kb":35120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":949407,"num_keys_recovered":27,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.1b","version":"215'418","reported_seq":4069,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.848781+0000","last_change":"2024-04-24T00:37:27.848781+0000","last_active":"2024-04-24T00:37:27.848781+0000","last_peered":"2024-04-24T00:37:27.848781+0000","last_clean":"2024-04-24T00:37:27.848781+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:37:27.848781+0000","last_undegraded":"2024-04-24T00:37:27.848781+0000","last_fullsized":"2024-04-24T00:37:27.848781+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:31:17.270241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032264390000000002,"stat_sum":{"num_bytes":43140017,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3956,"num_read_kb":45430,"num_write":528,"num_write_kb":27378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.1a","version":"215'338","reported_seq":5234,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.060061+0000","last_change":"2024-04-24T00:37:27.853142+0000","last_active":"2024-04-24T00:37:28.060061+0000","last_peered":"2024-04-24T00:37:28.060061+0000","last_clean":"2024-04-24T00:37:28.060061+0000","last_became_active":"2024-04-24T00:35:47.582331+0000","last_became_peered":"2024-04-24T00:35:47.582331+0000","last_unstale":"2024-04-24T00:37:28.060061+0000","last_undegraded":"2024-04-24T00:37:28.060061+0000","last_fullsized":"2024-04-24T00:37:28.060061+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:16:59.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027924579999999998,"stat_sum":{"num_bytes":20094273,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":4263,"num_read_kb":37095,"num_write":447,"num_write_kb":25754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":1449843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.19","version":"215'656","reported_seq":5508,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.032032+0000","last_change":"2024-04-24T00:37:27.848171+0000","last_active":"2024-04-24T00:37:28.032032+0000","last_peered":"2024-04-24T00:37:28.032032+0000","last_clean":"2024-04-24T00:37:28.032032+0000","last_became_active":"2024-04-24T00:35:47.589674+0000","last_became_peered":"2024-04-24T00:35:47.589674+0000","last_unstale":"2024-04-24T00:37:28.032032+0000","last_undegraded":"2024-04-24T00:37:28.032032+0000","last_fullsized":"2024-04-24T00:37:28.032032+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:33:44.310644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017102630000000001,"stat_sum":{"num_bytes":39133539,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":5921,"num_read_kb":40379,"num_write":951,"num_write_kb":29815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":868041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.18","version":"214'495","reported_seq":4470,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.891858+0000","last_change":"2024-04-24T00:37:27.892145+0000","last_active":"2024-04-24T00:37:27.891858+0000","last_peered":"2024-04-24T00:37:27.891858+0000","last_clean":"2024-04-24T00:37:27.891858+0000","last_became_active":"2024-04-24T00:36:56.588598+0000","last_became_peered":"2024-04-24T00:36:56.588598+0000","last_unstale":"2024-04-24T00:37:27.891858+0000","last_undegraded":"2024-04-24T00:37:27.891858+0000","last_fullsized":"2024-04-24T00:37:27.891858+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:14:32.596256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051444200000000005,"stat_sum":{"num_bytes":40542091,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":5187,"num_read_kb":36737,"num_write":548,"num_write_kb":26567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":17435009,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.17","version":"215'300","reported_seq":4451,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.850763+0000","last_change":"2024-04-24T00:37:27.850763+0000","last_active":"2024-04-24T00:37:27.850763+0000","last_peered":"2024-04-24T00:37:27.850763+0000","last_clean":"2024-04-24T00:37:27.850763+0000","last_became_active":"2024-04-24T00:36:55.593526+0000","last_became_peered":"2024-04-24T00:36:55.593526+0000","last_unstale":"2024-04-24T00:37:27.850763+0000","last_undegraded":"2024-04-24T00:37:27.850763+0000","last_fullsized":"2024-04-24T00:37:27.850763+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:13:31.794708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047401910000000004,"stat_sum":{"num_bytes":24936070,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":395,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":14291010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.16","version":"215'640","reported_seq":6166,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.847167+0000","last_change":"2024-04-24T00:37:27.847167+0000","last_active":"2024-04-24T00:37:27.847167+0000","last_peered":"2024-04-24T00:37:27.847167+0000","last_clean":"2024-04-24T00:37:27.847167+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:37:27.847167+0000","last_undegraded":"2024-04-24T00:37:27.847167+0000","last_fullsized":"2024-04-24T00:37:27.847167+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:28.720857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035607899999999998,"stat_sum":{"num_bytes":66050218,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":7145,"num_read_kb":50535,"num_write":993,"num_write_kb":38907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.15","version":"215'570","reported_seq":6576,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.846923+0000","last_change":"2024-04-24T00:37:27.846923+0000","last_active":"2024-04-24T00:37:27.846923+0000","last_peered":"2024-04-24T00:37:27.846923+0000","last_clean":"2024-04-24T00:37:27.846923+0000","last_became_active":"2024-04-24T00:35:28.725126+0000","last_became_peered":"2024-04-24T00:35:28.725126+0000","last_unstale":"2024-04-24T00:37:27.846923+0000","last_undegraded":"2024-04-24T00:37:27.846923+0000","last_fullsized":"2024-04-24T00:37:27.846923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:55:34.262347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042073000000000002,"stat_sum":{"num_bytes":69704437,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":6302,"num_read_kb":55518,"num_write":840,"num_write_kb":31179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":2312722,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.14","version":"215'446","reported_seq":4830,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.010171+0000","last_change":"2024-04-24T00:37:27.855175+0000","last_active":"2024-04-24T00:37:28.010171+0000","last_peered":"2024-04-24T00:37:28.010171+0000","last_clean":"2024-04-24T00:37:28.010171+0000","last_became_active":"2024-04-24T00:35:28.722992+0000","last_became_peered":"2024-04-24T00:35:28.722992+0000","last_unstale":"2024-04-24T00:37:28.010171+0000","last_undegraded":"2024-04-24T00:37:28.010171+0000","last_fullsized":"2024-04-24T00:37:28.010171+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:01.208016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032021689999999999,"stat_sum":{"num_bytes":50266964,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":4109,"num_read_kb":41556,"num_write":533,"num_write_kb":30977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1929866,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"2.7","version":"0'0","reported_seq":407,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.860547+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:37:27.860547+0000","last_peered":"2024-04-24T00:37:27.860547+0000","last_clean":"2024-04-24T00:37:27.860547+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:37:27.860547+0000","last_undegraded":"2024-04-24T00:37:27.860547+0000","last_fullsized":"2024-04-24T00:37:27.860547+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"215'498","reported_seq":5883,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.855862+0000","last_change":"2024-04-24T00:37:27.855862+0000","last_active":"2024-04-24T00:37:27.855862+0000","last_peered":"2024-04-24T00:37:27.855862+0000","last_clean":"2024-04-24T00:37:27.855862+0000","last_became_active":"2024-04-24T00:36:56.589104+0000","last_became_peered":"2024-04-24T00:36:56.589104+0000","last_unstale":"2024-04-24T00:37:27.855862+0000","last_undegraded":"2024-04-24T00:37:27.855862+0000","last_fullsized":"2024-04-24T00:37:27.855862+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:43:52.208931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034001080000000002,"stat_sum":{"num_bytes":66357231,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":5109,"num_read_kb":58055,"num_write":857,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":34869608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"2.6","version":"0'0","reported_seq":199,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.860609+0000","last_change":"2024-04-24T00:35:44.571029+0000","last_active":"2024-04-24T00:37:27.860609+0000","last_peered":"2024-04-24T00:37:27.860609+0000","last_clean":"2024-04-24T00:37:27.860609+0000","last_became_active":"2024-04-24T00:35:44.570229+0000","last_became_peered":"2024-04-24T00:35:44.570229+0000","last_unstale":"2024-04-24T00:37:27.860609+0000","last_undegraded":"2024-04-24T00:37:27.860609+0000","last_fullsized":"2024-04-24T00:37:27.860609+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:49:14.935258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"215'558","reported_seq":4388,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.000642+0000","last_change":"2024-04-24T00:37:27.848091+0000","last_active":"2024-04-24T00:37:28.000642+0000","last_peered":"2024-04-24T00:37:28.000642+0000","last_clean":"2024-04-24T00:37:28.000642+0000","last_became_active":"2024-04-24T00:36:56.590000+0000","last_became_peered":"2024-04-24T00:36:56.590000+0000","last_unstale":"2024-04-24T00:37:28.000642+0000","last_undegraded":"2024-04-24T00:37:28.000642+0000","last_fullsized":"2024-04-24T00:37:28.000642+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:53:34.261271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027094630000000001,"stat_sum":{"num_bytes":64767363,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":4335,"num_read_kb":41655,"num_write":788,"num_write_kb":26241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":24238018,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"2.5","version":"0'0","reported_seq":277,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.852068+0000","last_change":"2024-04-24T00:36:52.603941+0000","last_active":"2024-04-24T00:37:27.852068+0000","last_peered":"2024-04-24T00:37:27.852068+0000","last_clean":"2024-04-24T00:37:27.852068+0000","last_became_active":"2024-04-24T00:36:52.603227+0000","last_became_peered":"2024-04-24T00:36:52.603227+0000","last_unstale":"2024-04-24T00:37:27.852068+0000","last_undegraded":"2024-04-24T00:37:27.852068+0000","last_fullsized":"2024-04-24T00:37:27.852068+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:00:34.757330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"215'461","reported_seq":3680,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.892832+0000","last_change":"2024-04-24T00:37:27.892832+0000","last_active":"2024-04-24T00:37:27.892832+0000","last_peered":"2024-04-24T00:37:27.892832+0000","last_clean":"2024-04-24T00:37:27.892832+0000","last_became_active":"2024-04-24T00:35:47.586287+0000","last_became_peered":"2024-04-24T00:35:47.586287+0000","last_unstale":"2024-04-24T00:37:27.892832+0000","last_undegraded":"2024-04-24T00:37:27.892832+0000","last_fullsized":"2024-04-24T00:37:27.892832+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:55:34.948939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00109381,"stat_sum":{"num_bytes":31607790,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":3263,"num_read_kb":34258,"num_write":672,"num_write_kb":26457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1579365,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.891536+0000","last_change":"2024-04-24T00:36:52.599321+0000","last_active":"2024-04-24T00:37:27.891536+0000","last_peered":"2024-04-24T00:37:27.891536+0000","last_clean":"2024-04-24T00:37:27.891536+0000","last_became_active":"2024-04-24T00:36:52.598961+0000","last_became_peered":"2024-04-24T00:36:52.598961+0000","last_unstale":"2024-04-24T00:37:27.891536+0000","last_undegraded":"2024-04-24T00:37:27.891536+0000","last_fullsized":"2024-04-24T00:37:27.891536+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:25:49.716424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"215'541","reported_seq":5614,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.851483+0000","last_change":"2024-04-24T00:37:27.851483+0000","last_active":"2024-04-24T00:37:27.851483+0000","last_peered":"2024-04-24T00:37:27.851483+0000","last_clean":"2024-04-24T00:37:27.851483+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:37:27.851483+0000","last_undegraded":"2024-04-24T00:37:27.851483+0000","last_fullsized":"2024-04-24T00:37:27.851483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:31:17.013148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004132152,"stat_sum":{"num_bytes":51223636,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":5617,"num_read_kb":55178,"num_write":731,"num_write_kb":40829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.3","version":"215'375","reported_seq":5653,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.047174+0000","last_change":"2024-04-24T00:37:27.854555+0000","last_active":"2024-04-24T00:37:28.047174+0000","last_peered":"2024-04-24T00:37:28.047174+0000","last_clean":"2024-04-24T00:37:28.047174+0000","last_became_active":"2024-04-24T00:36:55.592083+0000","last_became_peered":"2024-04-24T00:36:55.592083+0000","last_unstale":"2024-04-24T00:37:28.047174+0000","last_undegraded":"2024-04-24T00:37:28.047174+0000","last_fullsized":"2024-04-24T00:37:28.047174+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:56.784188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032425790000000002,"stat_sum":{"num_bytes":44363007,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":4671,"num_read_kb":51115,"num_write":544,"num_write_kb":30767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":28550722,"num_keys_recovered":22,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"2.2","version":"0'0","reported_seq":333,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.844666+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:37:27.844666+0000","last_peered":"2024-04-24T00:37:27.844666+0000","last_clean":"2024-04-24T00:37:27.844666+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:37:27.844666+0000","last_undegraded":"2024-04-24T00:37:27.844666+0000","last_fullsized":"2024-04-24T00:37:27.844666+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"215'541","reported_seq":7310,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.892176+0000","last_change":"2024-04-24T00:37:27.892176+0000","last_active":"2024-04-24T00:37:27.892176+0000","last_peered":"2024-04-24T00:37:27.892176+0000","last_clean":"2024-04-24T00:37:27.892176+0000","last_became_active":"2024-04-24T00:35:47.586354+0000","last_became_peered":"2024-04-24T00:35:47.586354+0000","last_unstale":"2024-04-24T00:37:27.892176+0000","last_undegraded":"2024-04-24T00:37:27.892176+0000","last_fullsized":"2024-04-24T00:37:27.892176+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:17.049454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044640100000000002,"stat_sum":{"num_bytes":52338790,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":7093,"num_read_kb":65573,"num_write":625,"num_write_kb":34848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3429938,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"2.1","version":"0'0","reported_seq":402,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.891479+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:37:27.891479+0000","last_peered":"2024-04-24T00:37:27.891479+0000","last_clean":"2024-04-24T00:37:27.891479+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:37:27.891479+0000","last_undegraded":"2024-04-24T00:37:27.891479+0000","last_fullsized":"2024-04-24T00:37:27.891479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"215'407","reported_seq":3834,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.009330+0000","last_change":"2024-04-24T00:37:27.846260+0000","last_active":"2024-04-24T00:37:28.009330+0000","last_peered":"2024-04-24T00:37:28.009330+0000","last_clean":"2024-04-24T00:37:28.009330+0000","last_became_active":"2024-04-24T00:36:57.585413+0000","last_became_peered":"2024-04-24T00:36:57.585413+0000","last_unstale":"2024-04-24T00:37:28.009330+0000","last_undegraded":"2024-04-24T00:37:28.009330+0000","last_fullsized":"2024-04-24T00:37:28.009330+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:06.785075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001592085,"stat_sum":{"num_bytes":67290911,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2119,"num_read_kb":33775,"num_write":524,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":23613684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.846344+0000","last_change":"2024-04-24T00:36:52.602371+0000","last_active":"2024-04-24T00:37:27.846344+0000","last_peered":"2024-04-24T00:37:27.846344+0000","last_clean":"2024-04-24T00:37:27.846344+0000","last_became_active":"2024-04-24T00:36:52.601918+0000","last_became_peered":"2024-04-24T00:36:52.601918+0000","last_unstale":"2024-04-24T00:37:27.846344+0000","last_undegraded":"2024-04-24T00:37:27.846344+0000","last_fullsized":"2024-04-24T00:37:27.846344+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:07:14.214018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"215'528","reported_seq":2406,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.992904+0000","last_change":"2024-04-24T00:37:27.847159+0000","last_active":"2024-04-24T00:37:27.992904+0000","last_peered":"2024-04-24T00:37:27.992904+0000","last_clean":"2024-04-24T00:37:27.992904+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:37:27.992904+0000","last_undegraded":"2024-04-24T00:37:27.992904+0000","last_fullsized":"2024-04-24T00:37:27.992904+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:06:53.596921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001966242,"stat_sum":{"num_bytes":47919737,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":2520,"num_read_kb":24120,"num_write":702,"num_write_kb":27933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"2.3","version":"0'0","reported_seq":333,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.844929+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:37:27.844929+0000","last_peered":"2024-04-24T00:37:27.844929+0000","last_clean":"2024-04-24T00:37:27.844929+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:37:27.844929+0000","last_undegraded":"2024-04-24T00:37:27.844929+0000","last_fullsized":"2024-04-24T00:37:27.844929+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":459,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.860944+0000","last_change":"2024-04-24T00:36:55.589554+0000","last_active":"2024-04-24T00:37:27.860944+0000","last_peered":"2024-04-24T00:37:27.860944+0000","last_clean":"2024-04-24T00:37:27.860944+0000","last_became_active":"2024-04-24T00:36:55.587498+0000","last_became_peered":"2024-04-24T00:36:55.587498+0000","last_unstale":"2024-04-24T00:37:27.860944+0000","last_undegraded":"2024-04-24T00:37:27.860944+0000","last_fullsized":"2024-04-24T00:37:27.860944+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:33:25.475587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"215'830","reported_seq":6465,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.091317+0000","last_change":"2024-04-24T00:37:27.852139+0000","last_active":"2024-04-24T00:37:28.091317+0000","last_peered":"2024-04-24T00:37:28.091317+0000","last_clean":"2024-04-24T00:37:28.091317+0000","last_became_active":"2024-04-24T00:35:54.570587+0000","last_became_peered":"2024-04-24T00:35:54.570587+0000","last_unstale":"2024-04-24T00:37:28.091317+0000","last_undegraded":"2024-04-24T00:37:28.091317+0000","last_fullsized":"2024-04-24T00:37:28.091317+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:30:41.774011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027726890000000001,"stat_sum":{"num_bytes":67475443,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":7204,"num_read_kb":68771,"num_write":1150,"num_write_kb":32528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":5566512,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.9","version":"215'860","reported_seq":4445,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.946674+0000","last_change":"2024-04-24T00:37:27.855009+0000","last_active":"2024-04-24T00:37:27.946674+0000","last_peered":"2024-04-24T00:37:27.946674+0000","last_clean":"2024-04-24T00:37:27.946674+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:37:27.946674+0000","last_undegraded":"2024-04-24T00:37:27.946674+0000","last_fullsized":"2024-04-24T00:37:27.946674+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:33:25.450938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035487230000000002,"stat_sum":{"num_bytes":50256611,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":6708,"num_read_kb":32241,"num_write":1130,"num_write_kb":21131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.a","version":"215'1818","reported_seq":3932,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.085644+0000","last_change":"2024-04-24T00:37:27.864017+0000","last_active":"2024-04-24T00:37:28.085644+0000","last_peered":"2024-04-24T00:37:28.085644+0000","last_clean":"2024-04-24T00:37:28.085644+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:37:28.085644+0000","last_undegraded":"2024-04-24T00:37:28.085644+0000","last_fullsized":"2024-04-24T00:37:28.085644+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1818,"log_dups_size":0,"ondisk_log_size":1818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:46:32.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032843920000000001,"stat_sum":{"num_bytes":50111302,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":9419,"num_read_kb":44822,"num_write":3167,"num_write_kb":33601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.b","version":"215'405","reported_seq":3906,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.996866+0000","last_change":"2024-04-24T00:37:27.853853+0000","last_active":"2024-04-24T00:37:27.996866+0000","last_peered":"2024-04-24T00:37:27.996866+0000","last_clean":"2024-04-24T00:37:27.996866+0000","last_became_active":"2024-04-24T00:36:55.589472+0000","last_became_peered":"2024-04-24T00:36:55.589472+0000","last_unstale":"2024-04-24T00:37:27.996866+0000","last_undegraded":"2024-04-24T00:37:27.996866+0000","last_fullsized":"2024-04-24T00:37:27.996866+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:45:56.025258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034481580000000002,"stat_sum":{"num_bytes":20192954,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":3810,"num_read_kb":34636,"num_write":424,"num_write_kb":21900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":13450690,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.c","version":"215'708","reported_seq":9510,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.014742+0000","last_change":"2024-04-24T00:37:27.847900+0000","last_active":"2024-04-24T00:37:28.014742+0000","last_peered":"2024-04-24T00:37:28.014742+0000","last_clean":"2024-04-24T00:37:28.014742+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:37:28.014742+0000","last_undegraded":"2024-04-24T00:37:28.014742+0000","last_fullsized":"2024-04-24T00:37:28.014742+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:04:03.420906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025911770000000001,"stat_sum":{"num_bytes":79458748,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":10157,"num_read_kb":68419,"num_write":1062,"num_write_kb":35192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.d","version":"215'399","reported_seq":2600,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.050766+0000","last_change":"2024-04-24T00:37:27.849851+0000","last_active":"2024-04-24T00:37:28.050766+0000","last_peered":"2024-04-24T00:37:28.050766+0000","last_clean":"2024-04-24T00:37:28.050766+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:37:28.050766+0000","last_undegraded":"2024-04-24T00:37:28.050766+0000","last_fullsized":"2024-04-24T00:37:28.050766+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:32.872364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003783378,"stat_sum":{"num_bytes":33770538,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1995,"num_read_kb":38256,"num_write":529,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.e","version":"215'603","reported_seq":7423,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.089582+0000","last_change":"2024-04-24T00:37:27.856417+0000","last_active":"2024-04-24T00:37:28.089582+0000","last_peered":"2024-04-24T00:37:28.089582+0000","last_clean":"2024-04-24T00:37:28.089582+0000","last_became_active":"2024-04-24T00:35:29.560215+0000","last_became_peered":"2024-04-24T00:35:29.560215+0000","last_unstale":"2024-04-24T00:37:28.089582+0000","last_undegraded":"2024-04-24T00:37:28.089582+0000","last_fullsized":"2024-04-24T00:37:28.089582+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:25:56.460611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003619017,"stat_sum":{"num_bytes":45730860,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":7706,"num_read_kb":68962,"num_write":889,"num_write_kb":38889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":2728103,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.f","version":"215'805","reported_seq":5057,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:28.001414+0000","last_change":"2024-04-24T00:37:27.855773+0000","last_active":"2024-04-24T00:37:28.001414+0000","last_peered":"2024-04-24T00:37:28.001414+0000","last_clean":"2024-04-24T00:37:28.001414+0000","last_became_active":"2024-04-24T00:36:57.587551+0000","last_became_peered":"2024-04-24T00:36:57.587551+0000","last_unstale":"2024-04-24T00:37:28.001414+0000","last_undegraded":"2024-04-24T00:37:28.001414+0000","last_fullsized":"2024-04-24T00:37:28.001414+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:34.591834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033194940000000001,"stat_sum":{"num_bytes":39165240,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":5473,"num_read_kb":46023,"num_write":1302,"num_write_kb":43850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":24346262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.10","version":"215'513","reported_seq":5632,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.864944+0000","last_change":"2024-04-24T00:37:27.864944+0000","last_active":"2024-04-24T00:37:27.864944+0000","last_peered":"2024-04-24T00:37:27.864944+0000","last_clean":"2024-04-24T00:37:27.864944+0000","last_became_active":"2024-04-24T00:36:55.589936+0000","last_became_peered":"2024-04-24T00:36:55.589936+0000","last_unstale":"2024-04-24T00:37:27.864944+0000","last_undegraded":"2024-04-24T00:37:27.864944+0000","last_fullsized":"2024-04-24T00:37:27.864944+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:24:33.304883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039854629999999999,"stat_sum":{"num_bytes":38937237,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":5800,"num_read_kb":30295,"num_write":622,"num_write_kb":20569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":13050791,"num_keys_recovered":55,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.11","version":"215'510","reported_seq":7775,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.992959+0000","last_change":"2024-04-24T00:37:27.853442+0000","last_active":"2024-04-24T00:37:27.992959+0000","last_peered":"2024-04-24T00:37:27.992959+0000","last_clean":"2024-04-24T00:37:27.992959+0000","last_became_active":"2024-04-24T00:35:28.722568+0000","last_became_peered":"2024-04-24T00:35:28.722568+0000","last_unstale":"2024-04-24T00:37:27.992959+0000","last_undegraded":"2024-04-24T00:37:27.992959+0000","last_fullsized":"2024-04-24T00:37:27.992959+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:42:09.377936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034619770000000002,"stat_sum":{"num_bytes":88483942,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":7207,"num_read_kb":84249,"num_write":614,"num_write_kb":47836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":1618865,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.12","version":"214'1409","reported_seq":5335,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.892589+0000","last_change":"2024-04-24T00:37:27.892822+0000","last_active":"2024-04-24T00:37:27.892589+0000","last_peered":"2024-04-24T00:37:27.892589+0000","last_clean":"2024-04-24T00:37:27.892589+0000","last_became_active":"2024-04-24T00:36:55.588500+0000","last_became_peered":"2024-04-24T00:36:55.588500+0000","last_unstale":"2024-04-24T00:37:27.892589+0000","last_undegraded":"2024-04-24T00:37:27.892589+0000","last_fullsized":"2024-04-24T00:37:27.892589+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1409,"log_dups_size":0,"ondisk_log_size":1409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:34:16.428026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087874899999999996,"stat_sum":{"num_bytes":47226019,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":7341,"num_read_kb":53744,"num_write":2191,"num_write_kb":43757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":394,"num_bytes_recovered":39551111,"num_keys_recovered":65,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]},{"pgid":"3.13","version":"215'586","reported_seq":4432,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-24T00:37:27.854174+0000","last_change":"2024-04-24T00:37:27.854174+0000","last_active":"2024-04-24T00:37:27.854174+0000","last_peered":"2024-04-24T00:37:27.854174+0000","last_clean":"2024-04-24T00:37:27.854174+0000","last_became_active":"2024-04-24T00:35:28.722941+0000","last_became_peered":"2024-04-24T00:35:28.722941+0000","last_unstale":"2024-04-24T00:37:27.854174+0000","last_undegraded":"2024-04-24T00:37:27.854174+0000","last_fullsized":"2024-04-24T00:37:27.854174+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:08.728067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031713919999999999,"stat_sum":{"num_bytes":42226907,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":4669,"num_read_kb":38994,"num_write":866,"num_write_kb":31076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":962033,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4c","length":"1"},{"start":"4e","length":"1"},{"start":"50","length":"6"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1636911426,"num_objects":1248,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":177614,"num_read_kb":1538656,"num_write":28718,"num_write_kb":1017675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":282588799,"num_keys_recovered":395,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":85770240,"data_stored":868797149,"data_compressed":18521863,"data_compressed_allocated":74891264,"data_compressed_original":863133785,"omap_allocated":4834,"internal_metadata":0},"log_size":20113,"ondisk_log_size":20113,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":34064,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574899,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84176,"kb_used_data":12732,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94287664,"statfs":{"total":96636764160,"available":96550567936,"internally_reserved":0,"allocated":13037568,"data_stored":120639829,"data_compressed":1666276,"data_compressed_allocated":9363456,"data_compressed_original":118831108,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66520,"kb_used_data":21300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94305320,"statfs":{"total":96636764160,"available":96568647680,"internally_reserved":0,"allocated":21811200,"data_stored":191387993,"data_compressed":4398062,"data_compressed_allocated":16760832,"data_compressed_original":188816319,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92380,"kb_used_data":20588,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94279460,"statfs":{"total":96636764160,"available":96542167040,"internally_reserved":0,"allocated":21082112,"data_stored":201034941,"data_compressed":3470203,"data_compressed_allocated":16351232,"data_compressed_original":198678843,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574899,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77076,"kb_used_data":13952,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94294764,"statfs":{"total":96636764160,"available":96557838336,"internally_reserved":0,"allocated":14286848,"data_stored":111343189,"data_compressed":2850715,"data_compressed_allocated":10067968,"data_compressed_original":109187262,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64804,"kb_used_data":11408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94307036,"statfs":{"total":96636764160,"available":96570404864,"internally_reserved":0,"allocated":11681792,"data_stored":90395733,"data_compressed":1991720,"data_compressed_allocated":7860224,"data_compressed_original":88477394,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":4920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":5038080,"data_stored":28140833,"data_compressed":400827,"data_compressed_allocated":2142208,"data_compressed_original":26691584,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":408021893148,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11362304,"data_stored":112474769,"data_compressed":2575970,"data_compressed_allocated":9867264,"data_compressed_original":111661748,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2322432,"data_stored":26315982,"data_compressed":381165,"data_compressed_allocated":2088960,"data_compressed_original":26189824,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9039872,"data_stored":88525282,"data_compressed":1984735,"data_compressed_allocated":7823360,"data_compressed_original":87887570,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11259904,"data_stored":107898504,"data_compressed":2765923,"data_compressed_allocated":9854976,"data_compressed_original":107176211,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18481152,"data_stored":199754858,"data_compressed":3470203,"data_compressed_allocated":16351232,"data_compressed_original":198678843,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19210240,"data_stored":190107910,"data_compressed":4398062,"data_compressed_allocated":16760832,"data_compressed_original":188816319,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":10395648,"data_stored":118769378,"data_compressed":1659291,"data_compressed_allocated":9326592,"data_compressed_original":118241284,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:30.501 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:30.501 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:30.501 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:30.501 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:30.501 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:30.506 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:30.506+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:30.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:30.607 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:30.603+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110941 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 32768~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.760553690000002, 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99518982899999997, 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110941 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882159+0000", 2024-04-24T00:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882161+0000", 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882169+0000", 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8480000000000001e-06 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882173+0000", 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6110000000000004e-06 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886136+0000", 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039630810000000002 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886147+0000", 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0484e-05 2024-04-24T00:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876102+0000", 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98995489400000003 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876434+0000", 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033270799999999997 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.877348+0000", 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091357700000000005 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110942 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 229376~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.760531944, 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99714419399999998, 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110942 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5199999999999998e-07 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882182+0000", 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8819999999999999e-06 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882190+0000", 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3629999999999996e-06 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882194+0000", 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5000000000000001e-06 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886176+0000", 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039815329999999998 2024-04-24T00:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886184+0000", 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878386+0000", 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99220202999999996 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878535+0000", 2024-04-24T00:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001494 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879324+0000", 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078909599999999998 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110958 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 65536~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.760435631999997, 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.998192368, 2024-04-24T00:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110958 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882277+0000", 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882278+0000", 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6419999999999999e-06 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882284+0000", 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3890000000000004e-06 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882288+0000", 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.138e-06 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886955+0000", 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046673289999999996 2024-04-24T00:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886964+0000", 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8310000000000007e-06 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879623+0000", 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99265892499999997 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879818+0000", 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019486800000000001 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880469+0000", 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065093099999999999 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110959 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 262144~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.760375734, 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99945926699999998, 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110959 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882337+0000", 2024-04-24T00:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4499999999999997e-07 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882340+0000", 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0259999999999999e-06 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882358+0000", 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8689999999999999e-05 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882365+0000", 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3879999999999997e-06 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886989+0000", 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046244930000000004 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886997+0000", 2024-04-24T00:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880603+0000", 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99360585199999996 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880779+0000", 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175792 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881795+0000", 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001016922 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110974 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 98304~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.760228288999997, 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.000617986, 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110974 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:30.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2100000000000002e-07 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882486+0000", 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.215e-06 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882494+0000", 2024-04-24T00:37:30.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9410000000000007e-06 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882500+0000", 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2389999999999999e-06 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888427+0000", 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059265029999999996 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888436+0000", 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2019999999999993e-06 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881936+0000", 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99350027399999996 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.882146+0000", 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020958500000000001 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883102+0000", 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095560600000000001 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110975 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 294912~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.760204807000001, 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002014129, 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110975 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882508+0000", 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882510+0000", 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.148e-06 2024-04-24T00:37:30.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882515+0000", 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7289999999999997e-06 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882520+0000", 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3290000000000004e-06 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888463+0000", 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059433669999999997 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888470+0000", 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.187e-06 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883245+0000", 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99477520799999997 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883497+0000", 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025132699999999997 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884521+0000", 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001024435 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110990 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 131072~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.760053669000001, 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0028161980000001, 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110990 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882659+0000", 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4299999999999998e-07 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882661+0000", 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-06 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882667+0000", 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.879e-06 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882671+0000", 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5349999999999998e-06 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889926+0000", 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072543970000000001 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889935+0000", 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7070000000000004e-06 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884669+0000", 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99473391700000002 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884841+0000", 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017212600000000001 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885474+0000", 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063308099999999996 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110991 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 327680~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.760034615999999, 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0041004259999999, 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110991 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.686 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882678+0000", 2024-04-24T00:37:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882684+0000", 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3219999999999998e-06 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882691+0000", 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882705+0000", 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3813e-05 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889972+0000", 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072669309999999999 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889981+0000", 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3059999999999998e-06 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885568+0000", 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99558754299999996 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885798+0000", 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023032799999999999 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886778+0000", 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097934600000000008 2024-04-24T00:37:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111006 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 163840~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.759898823999997, 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0048379629999999, 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111006 2024-04-24T00:37:30.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-04-24T00:37:30.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882815+0000", 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7549999999999999e-06 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882822+0000", 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0539999999999997e-06 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882827+0000", 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3619999999999999e-06 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890715+0000", 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078878809999999994 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890723+0000", 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6289999999999999e-06 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886854+0000", 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99613063000000002 2024-04-24T00:37:30.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887032+0000", 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017786200000000001 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887651+0000", 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061942900000000001 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111007 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 360448~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.759879394999999, 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061255339999999, 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111007 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0400000000000002e-07 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882835+0000", 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7129999999999999e-06 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882842+0000", 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1280000000000002e-06 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882846+0000", 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5600000000000004e-06 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890749+0000", 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079023110000000004 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890758+0000", 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2070000000000001e-06 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887756+0000", 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99699837099999999 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888019+0000", 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263014 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888958+0000", 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093882600000000003 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111032 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.729945639, 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97659819800000003, 2024-04-24T00:37:30.700 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111032 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912767+0000", 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912769+0000", 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6759999999999999e-06 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912777+0000", 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7659999999999998e-06 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912782+0000", 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0590000000000002e-06 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913091+0000", 2024-04-24T00:37:30.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030847099999999999 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913101+0000", 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889077+0000", 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.975975437 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889228+0000", 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015131699999999999 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889364+0000", 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013657900000000001 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111065 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.705 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.719150044000003, 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96634889000000002, 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111065 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923563+0000", 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7100000000000001e-07 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923566+0000", 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.658e-06 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923577+0000", 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268999999999999e-05 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923588+0000", 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1299e-05 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923668+0000", 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0012000000000006e-05 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923678+0000", 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9019999999999994e-06 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889478+0000", 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96580058899999999 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889697+0000", 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218285 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889911+0000", 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000214105 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.710 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111116 3.1f 3:f85dfb34:::rbd_data.115e397a9cd4.000000000000000c:head [write 0~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.707948602000002, 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96849046100000002, 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111116 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934766+0000", 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3580000000000001e-06 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935127+0000", 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036108199999999998 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935150+0000", 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2756999999999999e-05 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935159+0000", 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-04-24T00:37:30.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935249+0000", 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9470999999999998e-05 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935263+0000", 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4445e-05 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889988+0000", 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95472527200000001 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890286+0000", 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029795199999999998 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890670+0000", 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038387799999999999 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:30.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902981+0000", 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012310399 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903018+0000", 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903151+0000", 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013311700000000001 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903254+0000", 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010306699999999999 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111119 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.706968656999997, 2024-04-24T00:37:30.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96712422899999995, 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111119 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935744+0000", 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.206e-06 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935750+0000", 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.203e-06 2024-04-24T00:37:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935761+0000", 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1266e-05 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935766+0000", 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2830000000000001e-06 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935841+0000", 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4685999999999996e-05 2024-04-24T00:37:30.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935854+0000", 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3135e-05 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902400+0000", 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96654596800000003 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902638+0000", 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023839799999999999 2024-04-24T00:37:30.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902868+0000", 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229084 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111159 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.678009772999999, 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.94163226300000002, 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.722 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111159 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964704+0000", 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964711+0000", 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4710000000000001e-06 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964727+0000", 2024-04-24T00:37:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6416e-05 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964736+0000", 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9209999999999998e-06 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964809+0000", 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3047999999999996e-05 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964821+0000", 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1738e-05 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.905869+0000", 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.941048035 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906136+0000", 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026686400000000001 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906334+0000", 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019838500000000001 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:123125 3.1f 3:facc60b2:::rbd_data.115ee312a896.000000000000001b:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.969648622999998, 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118275349, 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 123125 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673065+0000", 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673074+0000", 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1500000000000005e-06 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673099+0000", 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692e-05 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673119+0000", 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9683000000000001e-05 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673252+0000", 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133354 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673490+0000", 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023785700000000001 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.680775+0000", 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007284858 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791134+0000", 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110358923 2024-04-24T00:37:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791167+0000", 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2985999999999999e-05 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791260+0000", 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3060999999999996e-05 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791339+0000", 2024-04-24T00:37:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8844999999999995e-05 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:123125 3.1f e168/163 ondisk, result = 0)", 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.943762868, 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092199987999999997, 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698951+0000", 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7460000000000001e-06 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698959+0000", 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.746e-06 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698988+0000", 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9257000000000001e-05 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.699001+0000", 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3362e-05 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791078+0000", 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092076651999999995 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791120+0000", 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1789000000000003e-05 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791149+0000", 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9436000000000001e-05 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143625 3.18 3:1b6251e3:::rbd_data.115e8dbf22aa.0000000000000039:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.988067700000002, 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033012196000000001, 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143625 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654647+0000", 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654660+0000", 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.309e-05 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654691+0000", 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0709e-05 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654717+0000", 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6267000000000001e-05 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654859+0000", 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014176400000000001 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.655117+0000", 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025819200000000002 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.656922+0000", 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00180528 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687423+0000", 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030501117000000001 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687502+0000", 2024-04-24T00:37:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9127000000000003e-05 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687584+0000", 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2139999999999996e-05 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687656+0000", 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1989999999999993e-05 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143678 3.b 3:d70c067f:::rbd_data.115e8dbf22aa.000000000000005e:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.805779950000002, 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18713334100000001, 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143678 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836936+0000", 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9829999999999998e-06 2024-04-24T00:37:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836953+0000", 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7016000000000001e-05 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836986+0000", 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3319999999999999e-05 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837001+0000", 2024-04-24T00:37:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837183+0000", 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018131700000000001 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837430+0000", 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024749599999999999 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.847946+0000", 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01051566 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023623+0000", 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17567688400000001 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023871+0000", 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024814800000000001 2024-04-24T00:37:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023984+0000", 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011326499999999999 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024065+0000", 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1129000000000002e-05 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:143678 3.b e183/160 ondisk, result = 0)", 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.760373649999998, 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14174242100000001, 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882340+0000", 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882350+0000", 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882384+0000", 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3923e-05 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882407+0000", 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2619000000000001e-05 2024-04-24T00:37:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023712+0000", 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.141305088 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023728+0000", 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.607e-05 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024081+0000", 2024-04-24T00:37:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352648 2024-04-24T00:37:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:30.753 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:30.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:30.755 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:30.750+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:30.855 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:30.851+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:30.861 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:37:30.938 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:30.938 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:30.938 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:30.938 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:30.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:30.956 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:30.950+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:31.057 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:31.054+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:31.081 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:31.081 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:31.081 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:31.081 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:31.081 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:31.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:31.158 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:31.155+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.786639202, 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:37:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:37:31.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:37:31.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:31.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.731127716, 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:37:31.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:37:31.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.63055106799999, 2024-04-24T00:37:31.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:37:31.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:37:31.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:37:31.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.241 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 164.58898611399999, 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:37:31.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.02525455200001, 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.246 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:37:31.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:37:31.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.252 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.522281155, 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:37:31.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.283966922, 2024-04-24T00:37:31.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:37:31.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.249026499, 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.249005626, 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:37:31.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:37:31.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.248982247, 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.272 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:37:31.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.248960705, 2024-04-24T00:37:31.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.248939185, 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:37:31.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.248893028, 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.285 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.248868909, 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.24884550500001, 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:37:31.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.24881773200001, 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.24879631899999, 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:37:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:37:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.248775728, 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:37:31.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 112.032705486, 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:37:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:37:31.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:31.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 112.004734582, 2024-04-24T00:37:31.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:31.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.322 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:31.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:31.324 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:31.318+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:31.424 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:31.419+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:31.496 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:31.497 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:31.497 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:31.497 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:31.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:31.526 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:31.522+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:31.627 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:31.622+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:31.672 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:31.673 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:31.673 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:31.673 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:31.673 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:31.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:31.727 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:31.722+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:31.817 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 71.572513158000007, 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:37:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 71.542745676999999, 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:37:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126791 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 112640~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.173180647999999, 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074437884999999995, 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126791 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640034+0000", 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640044+0000", 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2660000000000007e-06 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640050+0000", 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713112+0000", 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073062031999999999 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713345+0000", 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000232616 2024-04-24T00:37:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714469+0000", 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011237669999999999 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126792 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 116736~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.173151875000002, 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075734135999999994, 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126792 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3399999999999999e-07 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640063+0000", 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7e-06 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640072+0000", 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0729999999999994e-06 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640084+0000", 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2486e-05 2024-04-24T00:37:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714588+0000", 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074503102000000002 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714763+0000", 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175885 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715794+0000", 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001030256 2024-04-24T00:37:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126793 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 120832~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.173118084999999, 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076915124000000001, 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126793 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640094+0000", 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640097+0000", 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9799999999999998e-06 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640107+0000", 2024-04-24T00:37:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6229999999999992e-06 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640123+0000", 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5892999999999998e-05 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715884+0000", 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075761384000000001 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.716061+0000", 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176727 2024-04-24T00:37:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717009+0000", 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094787400000000003 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126794 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 124928~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.173079405999999, 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078057396000000001, 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126794 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640133+0000", 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-07 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640136+0000", 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1e-06 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640145+0000", 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.516e-06 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640152+0000", 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717088+0000", 2024-04-24T00:37:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076936302999999998 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717201+0000", 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112704 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718189+0000", 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098839299999999995 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126795 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 129024~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.173050519999997, 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079207986999999994, 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126795 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640162+0000", 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7400000000000003e-07 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640165+0000", 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1860000000000001e-06 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640174+0000", 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5429999999999999e-06 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640181+0000", 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8029999999999999e-06 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718272+0000", 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078091277000000001 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718443+0000", 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171027 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.719369+0000", 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092547699999999996 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127300 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 229376~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.150794261000001, 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073789773000000003, 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127300 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662418+0000", 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.203e-06 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662439+0000", 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0713e-05 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662460+0000", 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.105e-05 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662488+0000", 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7966e-05 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735368+0000", 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072880167999999995 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735555+0000", 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018692399999999999 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736207+0000", 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065174899999999997 2024-04-24T00:37:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127301 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 233472~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.150693302000001, 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074400210999999994, 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127301 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662520+0000", 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8700000000000001e-06 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662527+0000", 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7830000000000001e-06 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662549+0000", 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2379e-05 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662564+0000", 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.473e-05 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736281+0000", 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073717486999999998 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736386+0000", 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104594 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736918+0000", 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053236799999999997 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127302 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.150625376999997, 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075297446000000004, 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127302 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662587+0000", 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096e-06 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662608+0000", 2024-04-24T00:37:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662624+0000", 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5563e-05 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662639+0000", 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4766999999999999e-05 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737013+0000", 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074374061000000005 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737195+0000", 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182201 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737884+0000", 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068866800000000003 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127303 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.150551499000002, 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075989524000000003, 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127303 2024-04-24T00:37:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662662+0000", 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5400000000000001e-06 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662668+0000", 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1970000000000001e-06 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662689+0000", 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0953000000000001e-05 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662703+0000", 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737962+0000", 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075258932000000001 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738074+0000", 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011173800000000001 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738649+0000", 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057545399999999998 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127304 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.150487165999998, 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076894431999999999, 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127304 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662725+0000", 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0359999999999999e-06 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662732+0000", 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6440000000000003e-06 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662751+0000", 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8583000000000001e-05 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662768+0000", 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7336000000000001e-05 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738744+0000", 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075976005999999999 2024-04-24T00:37:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738924+0000", 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017996900000000001 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739619+0000", 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069485800000000002 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127305 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.150422890999998, 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077646889999999996, 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127305 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662790+0000", 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662796+0000", 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3659999999999997e-06 2024-04-24T00:37:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662819+0000", 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2503000000000001e-05 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662834+0000", 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739755+0000", 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076920905999999997 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739876+0000", 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121445 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740435+0000", 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055911299999999997 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127306 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.150355328000003, 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078485103000000001, 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127306 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662857+0000", 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1599999999999999e-06 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662864+0000", 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3829999999999997e-06 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662883+0000", 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9476000000000001e-05 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662897+0000", 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3485999999999999e-05 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740564+0000", 2024-04-24T00:37:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077667768999999998 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740736+0000", 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017116700000000001 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741341+0000", 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060566199999999998 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127307 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.150295923000002, 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079342509000000006, 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127307 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662917+0000", 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662922+0000", 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662939+0000", 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833e-05 2024-04-24T00:37:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662954+0000", 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741518+0000", 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078564118000000002 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741709+0000", 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191337 2024-04-24T00:37:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.742258+0000", 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054904499999999996 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127559 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 243712~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.129511618000002, 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073184308000000003, 2024-04-24T00:37:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127559 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683701+0000", 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2300000000000001e-06 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683724+0000", 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.321e-05 2024-04-24T00:37:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683745+0000", 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.101e-05 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683761+0000", 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756124+0000", 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072363532999999994 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756327+0000", 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020239999999999999 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756884+0000", 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055760300000000005 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127560 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 247808~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.129427878999998, 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073924043999999994, 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127560 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683785+0000", 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1829999999999999e-06 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683792+0000", 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2799999999999998e-06 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683810+0000", 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8076000000000001e-05 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683842+0000", 2024-04-24T00:37:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756969+0000", 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073126925999999995 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757172+0000", 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020269900000000001 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757708+0000", 2024-04-24T00:37:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053609999999999997 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127561 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 251904~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.129346550999998, 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074620636000000004, 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127561 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683866+0000", 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.127e-06 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683873+0000", 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683893+0000", 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9300000000000002e-05 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683954+0000", 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1148999999999993e-05 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757783+0000", 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073829025000000006 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757939+0000", 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582700000000001 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758486+0000", 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054703200000000003 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127562 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 256000~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.129224956000002, 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075112683, 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127562 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683988+0000", 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.443e-06 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684000+0000", 2024-04-24T00:37:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.252e-05 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684025+0000", 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4956e-05 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684044+0000", 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8770000000000002e-05 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758566+0000", 2024-04-24T00:37:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074522094999999997 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758669+0000", 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010247399999999999 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759099+0000", 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043042500000000001 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127563 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 260096~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.129131123999997, 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075791184999999997, 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127563 2024-04-24T00:37:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684082+0000", 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684089+0000", 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4889999999999998e-06 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684115+0000", 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5643000000000002e-05 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684133+0000", 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7842999999999998e-05 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759174+0000", 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075041103999999997 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759329+0000", 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015504200000000001 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759872+0000", 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054272399999999996 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:31.911 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:31.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:31.913 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:31.910+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:32.013 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:32.010+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:171603 3.c e219/56 3:363a0967:::rbd_header.115e488d3b50:head v 219'728, mlcod=219'726)", 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:32.045188+0000", 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00044267500000000001, 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048166099999999998, 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:32.045188+0000", 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:32.045188+0000", 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:32.045190+0000", 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.595e-06 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:32.045201+0000", 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0705e-05 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:32.045219+0000", 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8394e-05 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:32.045233+0000", 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3617e-05 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:32.045364+0000", 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131137 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:32.045410+0000", 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6102e-05 2024-04-24T00:37:32.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.051 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:37:32.051 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-04-24T00:37:32.051 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:32.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:32.114 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:32.111+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:32.189 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:32.189 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:32.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:32.246 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:32.243+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:32.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:32.315 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:32.310+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:32.374 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:32.374 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:32.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:32.374 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:32.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 199.15902631200001, 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:37:32.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:37:32.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:32.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.380 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 199.158675397, 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:37:32.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:37:32.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:37:32.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:32.386 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 199.15178707699999, 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:32.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.89350545100001, 2024-04-24T00:37:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:37:32.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:37:32.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.883229035, 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.42127823200001, 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.395244211, 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:37:32.406 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:37:32.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.181301304, 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.17128936, 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:37:32.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:37:32.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.897910498, 2024-04-24T00:37:32.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:37:32.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:37:32.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.88124204899999, 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.69534426600001, 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.429 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:37:32.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:37:32.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:37:32.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:37:32.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.011585909000004, 2024-04-24T00:37:32.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:37:32.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:37:32.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:37:32.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 89.989368911, 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.684799072999994, 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:37:32.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:37:32.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.672174112999997, 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:32.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:37:32.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.322682625999999, 2024-04-24T00:37:32.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:37:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:37:32.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.307664430999999, 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:37:32.460 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.414848028, 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:37:32.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:37:32.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.377496001000001, 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:32.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:32.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:32.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:37:32.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:37:32.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.476 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:32.476 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:32.476 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:32.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:32.478 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:32.479 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:32.479 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:32.479 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:32.480 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:32.480 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:32.480 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:32.475+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:32.490 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:32.491 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:32.511 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:32.511 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:32.530 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:32.530 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:32.551 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:32.551 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:32.573 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:32.574 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:32.578 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:32.575+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:32.611 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:32.611 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:32.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:32.679 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:32.674+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:32.749 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:32.749 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:32.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:32.780 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:32.775+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:32.880 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:32.878+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:32.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:37:32.940 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:32.940 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:32.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:32.981 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:32.979+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:33.083 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:33.078+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:33.131 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:33.132 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:33.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:33.184 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:33.178+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:33.285 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:33.282+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:33.308 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:33.309 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:33.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:33.387 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:33.382+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:33.448 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:33.448 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:33.462 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:33.504 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:33.502+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:33.589 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:33.586+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:33.628 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:33.628 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:33.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:33.690 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:33.686+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:33.765 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:33.766 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:33.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:33.821 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:33.818+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:33.893 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:33.886+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:33.900 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:37:33.954 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:33.954 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:33.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:33.993 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:33.990+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:34.094 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:34.090+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:34.123 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:34.123 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:34.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:34.196 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:34.190+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:34.296 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:34.294+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:34.306 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:34.307 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:34.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:34.398 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:34.394+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:34.489 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:34.489 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:34.499 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:34.494+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:34.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:34.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:34.600 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:34.594+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:34.648 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:34.649 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:34.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:34.709 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:34.706+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:34.757 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:34.778 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:34.796 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:34.796 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:34.796 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:34.797 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:34.797 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:34.798 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:34.799 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:34.802 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:34.798+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:34.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:34.823 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:34.823 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:34.844 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:34.844 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:34.862 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:34.862 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:34.883 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:34.902 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:34.898+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:34.904 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:34.950 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:34.950 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:34.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:34.950 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:34.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110941 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 32768~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.065310431, 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99518982899999997, 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110941 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882159+0000", 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882161+0000", 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882169+0000", 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8480000000000001e-06 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882173+0000", 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6110000000000004e-06 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886136+0000", 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039630810000000002 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886147+0000", 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0484e-05 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876102+0000", 2024-04-24T00:37:34.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98995489400000003 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876434+0000", 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033270799999999997 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.877348+0000", 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091357700000000005 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110942 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 229376~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.065288684999999, 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99714419399999998, 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110942 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.956 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5199999999999998e-07 2024-04-24T00:37:34.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882182+0000", 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8819999999999999e-06 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882190+0000", 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3629999999999996e-06 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882194+0000", 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5000000000000001e-06 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886176+0000", 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039815329999999998 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886184+0000", 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878386+0000", 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99220202999999996 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878535+0000", 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001494 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879324+0000", 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078909599999999998 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110958 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 65536~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.065192373000002, 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.998192368, 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110958 2024-04-24T00:37:34.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882277+0000", 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882278+0000", 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6419999999999999e-06 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882284+0000", 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3890000000000004e-06 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882288+0000", 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.138e-06 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886955+0000", 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046673289999999996 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886964+0000", 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8310000000000007e-06 2024-04-24T00:37:34.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879623+0000", 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99265892499999997 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879818+0000", 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019486800000000001 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880469+0000", 2024-04-24T00:37:34.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065093099999999999 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110959 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 262144~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.065132474999999, 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99945926699999998, 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110959 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882337+0000", 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4499999999999997e-07 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882340+0000", 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0259999999999999e-06 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882358+0000", 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8689999999999999e-05 2024-04-24T00:37:34.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882365+0000", 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3879999999999997e-06 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886989+0000", 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046244930000000004 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886997+0000", 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-04-24T00:37:34.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880603+0000", 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99360585199999996 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880779+0000", 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175792 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881795+0000", 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001016922 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110974 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 98304~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.064985030000003, 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.000617986, 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110974 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2100000000000002e-07 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882486+0000", 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.215e-06 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882494+0000", 2024-04-24T00:37:34.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9410000000000007e-06 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882500+0000", 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2389999999999999e-06 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888427+0000", 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059265029999999996 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888436+0000", 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2019999999999993e-06 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881936+0000", 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99350027399999996 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.882146+0000", 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020958500000000001 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883102+0000", 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095560600000000001 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110975 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 294912~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.064961547999999, 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002014129, 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110975 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882508+0000", 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882510+0000", 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.148e-06 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882515+0000", 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7289999999999997e-06 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882520+0000", 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3290000000000004e-06 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888463+0000", 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059433669999999997 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888470+0000", 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.187e-06 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883245+0000", 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99477520799999997 2024-04-24T00:37:34.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883497+0000", 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025132699999999997 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884521+0000", 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001024435 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110990 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 131072~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.064810410000007, 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0028161980000001, 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110990 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.982 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882659+0000", 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4299999999999998e-07 2024-04-24T00:37:34.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882661+0000", 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-06 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882667+0000", 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.879e-06 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882671+0000", 2024-04-24T00:37:34.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5349999999999998e-06 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889926+0000", 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072543970000000001 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889935+0000", 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7070000000000004e-06 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884669+0000", 2024-04-24T00:37:34.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99473391700000002 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884841+0000", 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017212600000000001 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885474+0000", 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063308099999999996 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110991 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 327680~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.064791357000004, 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0041004259999999, 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110991 2024-04-24T00:37:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882678+0000", 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882684+0000", 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3219999999999998e-06 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882691+0000", 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882705+0000", 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3813e-05 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889972+0000", 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072669309999999999 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889981+0000", 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3059999999999998e-06 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885568+0000", 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99558754299999996 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885798+0000", 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023032799999999999 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886778+0000", 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097934600000000008 2024-04-24T00:37:34.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111006 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 163840~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.064655564999995, 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0048379629999999, 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111006 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882815+0000", 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7549999999999999e-06 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882822+0000", 2024-04-24T00:37:34.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0539999999999997e-06 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882827+0000", 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3619999999999999e-06 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890715+0000", 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078878809999999994 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890723+0000", 2024-04-24T00:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6289999999999999e-06 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886854+0000", 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99613063000000002 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887032+0000", 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017786200000000001 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887651+0000", 2024-04-24T00:37:34.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061942900000000001 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111007 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 360448~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.064636136000004, 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061255339999999, 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:34.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111007 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0400000000000002e-07 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882835+0000", 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7129999999999999e-06 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882842+0000", 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1280000000000002e-06 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:34.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882846+0000", 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5600000000000004e-06 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890749+0000", 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079023110000000004 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890758+0000", 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2070000000000001e-06 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887756+0000", 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99699837099999999 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888019+0000", 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263014 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888958+0000", 2024-04-24T00:37:35.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093882600000000003 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111032 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.034702379999999, 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97659819800000003, 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111032 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912767+0000", 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912769+0000", 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6759999999999999e-06 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912777+0000", 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7659999999999998e-06 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912782+0000", 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0590000000000002e-06 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913091+0000", 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030847099999999999 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913101+0000", 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889077+0000", 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.975975437 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889228+0000", 2024-04-24T00:37:35.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015131699999999999 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889364+0000", 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013657900000000001 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111065 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.023906784999994, 2024-04-24T00:37:35.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96634889000000002, 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111065 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923563+0000", 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7100000000000001e-07 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923566+0000", 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.658e-06 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923577+0000", 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268999999999999e-05 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923588+0000", 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1299e-05 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923668+0000", 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0012000000000006e-05 2024-04-24T00:37:35.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923678+0000", 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9019999999999994e-06 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889478+0000", 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96580058899999999 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889697+0000", 2024-04-24T00:37:35.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218285 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889911+0000", 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000214105 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111116 3.1f 3:f85dfb34:::rbd_data.115e397a9cd4.000000000000000c:head [write 0~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.012705342999993, 2024-04-24T00:37:35.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96849046100000002, 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111116 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934766+0000", 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3580000000000001e-06 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935127+0000", 2024-04-24T00:37:35.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036108199999999998 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935150+0000", 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2756999999999999e-05 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935159+0000", 2024-04-24T00:37:35.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935249+0000", 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9470999999999998e-05 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935263+0000", 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4445e-05 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889988+0000", 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95472527200000001 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890286+0000", 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029795199999999998 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890670+0000", 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038387799999999999 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902981+0000", 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012310399 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903018+0000", 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903151+0000", 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013311700000000001 2024-04-24T00:37:35.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903254+0000", 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010306699999999999 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111119 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.011725397999996, 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96712422899999995, 2024-04-24T00:37:35.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111119 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935744+0000", 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.206e-06 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935750+0000", 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.203e-06 2024-04-24T00:37:35.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935761+0000", 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1266e-05 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935766+0000", 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2830000000000001e-06 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935841+0000", 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4685999999999996e-05 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935854+0000", 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3135e-05 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902400+0000", 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96654596800000003 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902638+0000", 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023839799999999999 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902868+0000", 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229084 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111159 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:37:35.024 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.982766514000005, 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.94163226300000002, 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111159 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964704+0000", 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964711+0000", 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4710000000000001e-06 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964727+0000", 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6416e-05 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964736+0000", 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9209999999999998e-06 2024-04-24T00:37:35.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964809+0000", 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3047999999999996e-05 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964821+0000", 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1738e-05 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.905869+0000", 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.941048035 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906136+0000", 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026686400000000001 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906334+0000", 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019838500000000001 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:123125 3.1f 3:facc60b2:::rbd_data.115ee312a896.000000000000001b:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.274405364000003, 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118275349, 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 123125 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673063+0000", 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673065+0000", 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673074+0000", 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1500000000000005e-06 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673099+0000", 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692e-05 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673119+0000", 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9683000000000001e-05 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673252+0000", 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133354 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.673490+0000", 2024-04-24T00:37:35.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023785700000000001 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.680775+0000", 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.007284858 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791134+0000", 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110358923 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:35.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791167+0000", 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2985999999999999e-05 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791260+0000", 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3060999999999996e-05 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791339+0000", 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8844999999999995e-05 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:123125 3.1f e168/163 ondisk, result = 0)", 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.248519608999999, 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092199987999999997, 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698949+0000", 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698951+0000", 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7460000000000001e-06 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698959+0000", 2024-04-24T00:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.746e-06 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.698988+0000", 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9257000000000001e-05 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.699001+0000", 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3362e-05 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791078+0000", 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092076651999999995 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791120+0000", 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1789000000000003e-05 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:34.791149+0000", 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9436000000000001e-05 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143625 3.18 3:1b6251e3:::rbd_data.115e8dbf22aa.0000000000000039:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.292824441, 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033012196000000001, 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143625 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654644+0000", 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654647+0000", 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654660+0000", 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.309e-05 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654691+0000", 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0709e-05 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654717+0000", 2024-04-24T00:37:35.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6267000000000001e-05 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.654859+0000", 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014176400000000001 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.655117+0000", 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025819200000000002 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:35.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.656922+0000", 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00180528 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687423+0000", 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030501117000000001 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687502+0000", 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9127000000000003e-05 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687584+0000", 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2139999999999996e-05 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.687656+0000", 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1989999999999993e-05 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143678 3.b 3:d70c067f:::rbd_data.115e8dbf22aa.000000000000005e:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.110536691, 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18713334100000001, 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143678 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836936+0000", 2024-04-24T00:37:35.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9829999999999998e-06 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836953+0000", 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7016000000000001e-05 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836986+0000", 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3319999999999999e-05 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837001+0000", 2024-04-24T00:37:35.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837183+0000", 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018131700000000001 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837430+0000", 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024749599999999999 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:35.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.847946+0000", 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01051566 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023623+0000", 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17567688400000001 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023871+0000", 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024814800000000001 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023984+0000", 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011326499999999999 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024065+0000", 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1129000000000002e-05 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:143678 3.b e183/160 ondisk, result = 0)", 2024-04-24T00:37:35.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.065130390999997, 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14174242100000001, 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882338+0000", 2024-04-24T00:37:35.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882340+0000", 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.17e-06 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882350+0000", 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882384+0000", 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3923e-05 2024-04-24T00:37:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.882407+0000", 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2619000000000001e-05 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023712+0000", 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.141305088 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023728+0000", 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.607e-05 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024081+0000", 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352648 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.055 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:35.057 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:35.050+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:35.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:35.158 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:35.154+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:35.231 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:35.232 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:35.232 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:35.232 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:35.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:35.293 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:35.290+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:35.360 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:35.358+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:35.373 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:35.373 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:35.373 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:35.373 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:35.373 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:35.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:35.446 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:35.446 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:35.446 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-24T00:37:35.446 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:37:35.446 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:37:35.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-04-24T00:37:35.493 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:35.490+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 188.11475183499999, 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 188.05924034899999, 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 168.95866370100001, 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:37:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:37:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 168.91709874700001, 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 131.353367185, 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.85039378800001, 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:37:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.61207955499999, 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.577139132, 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.577118259, 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.57709488, 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:37:35.594 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:37:35.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.57707333800001, 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.577051818, 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.603 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.577005661, 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:35.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.611 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.576981542, 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:35.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.615 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.57695813799999, 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.576930365, 2024-04-24T00:37:35.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:37:35.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:37:35.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.576908952, 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.625 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:37:35.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:37:35.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:37:35.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.576888361, 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:37:35.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:37:35.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.360818119, 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:37:35.633 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:37:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:37:35.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:37:35.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:37:35.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.332847215, 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:37:35.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:37:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:35.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:35.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:35.644 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:35.645 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:35.646 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "" 2024-04-24T00:37:35.646 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:35.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:35.647 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:35.642+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:35.746 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:35.742+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:35.785 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:35.786 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:35.786 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:35.786 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:35.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:35.847 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:35.842+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:35.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:37:35.927 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:35.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:35.927 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:35.927 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:35.928 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:35.942 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:35.948 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:35.942+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:36.049 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:36.046+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:36.113 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.869143665999999, 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:37:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:37:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.839376185000006, 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:37:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:37:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126791 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 112640~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.469811155999999, 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074437884999999995, 2024-04-24T00:37:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126791 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640034+0000", 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640044+0000", 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2660000000000007e-06 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640050+0000", 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713112+0000", 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073062031999999999 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713345+0000", 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000232616 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714469+0000", 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011237669999999999 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126792 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 116736~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.469782383000002, 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075734135999999994, 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126792 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:36.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3399999999999999e-07 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640063+0000", 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7e-06 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640072+0000", 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0729999999999994e-06 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640084+0000", 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2486e-05 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714588+0000", 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074503102000000002 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714763+0000", 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175885 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715794+0000", 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001030256 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126793 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 120832~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.469748592999998, 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076915124000000001, 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126793 2024-04-24T00:37:36.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640094+0000", 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640097+0000", 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9799999999999998e-06 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640107+0000", 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6229999999999992e-06 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640123+0000", 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5892999999999998e-05 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715884+0000", 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075761384000000001 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.716061+0000", 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176727 2024-04-24T00:37:36.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717009+0000", 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094787400000000003 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126794 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 124928~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.469709913999999, 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078057396000000001, 2024-04-24T00:37:36.136 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126794 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640133+0000", 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-07 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640136+0000", 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1e-06 2024-04-24T00:37:36.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640145+0000", 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.516e-06 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640152+0000", 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717088+0000", 2024-04-24T00:37:36.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076936302999999998 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717201+0000", 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112704 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718189+0000", 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098839299999999995 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126795 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 129024~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.469681027999997, 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079207986999999994, 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126795 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.141 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640162+0000", 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7400000000000003e-07 2024-04-24T00:37:36.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640165+0000", 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1860000000000001e-06 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640174+0000", 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5429999999999999e-06 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640181+0000", 2024-04-24T00:37:36.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8029999999999999e-06 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718272+0000", 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078091277000000001 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718443+0000", 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171027 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.719369+0000", 2024-04-24T00:37:36.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092547699999999996 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127300 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 229376~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.447424769000001, 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073789773000000003, 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127300 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662418+0000", 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.203e-06 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662439+0000", 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0713e-05 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662460+0000", 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.105e-05 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662488+0000", 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7966e-05 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735368+0000", 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072880167999999995 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735555+0000", 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018692399999999999 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736207+0000", 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065174899999999997 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127301 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 233472~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.44732381, 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074400210999999994, 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.149 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127301 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:36.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662520+0000", 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8700000000000001e-06 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662527+0000", 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7830000000000001e-06 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662549+0000", 2024-04-24T00:37:36.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2379e-05 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662564+0000", 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.473e-05 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736281+0000", 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073717486999999998 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736386+0000", 2024-04-24T00:37:36.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104594 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736918+0000", 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053236799999999997 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127302 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.447255884999997, 2024-04-24T00:37:36.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075297446000000004, 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127302 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662587+0000", 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096e-06 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662608+0000", 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662624+0000", 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5563e-05 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662639+0000", 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4766999999999999e-05 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737013+0000", 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074374061000000005 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737195+0000", 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182201 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737884+0000", 2024-04-24T00:37:36.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068866800000000003 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127303 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.447182007000002, 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075989524000000003, 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127303 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:36.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662662+0000", 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5400000000000001e-06 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662668+0000", 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1970000000000001e-06 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662689+0000", 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0953000000000001e-05 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662703+0000", 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737962+0000", 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075258932000000001 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738074+0000", 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011173800000000001 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738649+0000", 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057545399999999998 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127304 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.162 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.447117673999998, 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076894431999999999, 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127304 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662725+0000", 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0359999999999999e-06 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662732+0000", 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6440000000000003e-06 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662751+0000", 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8583000000000001e-05 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662768+0000", 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7336000000000001e-05 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738744+0000", 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075976005999999999 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738924+0000", 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017996900000000001 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739619+0000", 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069485800000000002 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127305 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.447053398999998, 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077646889999999996, 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127305 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662790+0000", 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662796+0000", 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3659999999999997e-06 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662819+0000", 2024-04-24T00:37:36.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2503000000000001e-05 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662834+0000", 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739755+0000", 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076920905999999997 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739876+0000", 2024-04-24T00:37:36.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121445 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740435+0000", 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055911299999999997 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127306 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.446985836000003, 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078485103000000001, 2024-04-24T00:37:36.171 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127306 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662857+0000", 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1599999999999999e-06 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662864+0000", 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3829999999999997e-06 2024-04-24T00:37:36.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662883+0000", 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9476000000000001e-05 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662897+0000", 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3485999999999999e-05 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740564+0000", 2024-04-24T00:37:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077667768999999998 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740736+0000", 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017116700000000001 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741341+0000", 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060566199999999998 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127307 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.446926431000001, 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079342509000000006, 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127307 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662917+0000", 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:37:36.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662922+0000", 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662939+0000", 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833e-05 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662954+0000", 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:36.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741518+0000", 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078564118000000002 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741709+0000", 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191337 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.742258+0000", 2024-04-24T00:37:36.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054904499999999996 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127559 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 243712~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.426142126000002, 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073184308000000003, 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127559 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683701+0000", 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2300000000000001e-06 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683724+0000", 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.321e-05 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683745+0000", 2024-04-24T00:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.101e-05 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683761+0000", 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756124+0000", 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072363532999999994 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756327+0000", 2024-04-24T00:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020239999999999999 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756884+0000", 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055760300000000005 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127560 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 247808~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.426058386999998, 2024-04-24T00:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073924043999999994, 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127560 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683785+0000", 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1829999999999999e-06 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683792+0000", 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2799999999999998e-06 2024-04-24T00:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683810+0000", 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8076000000000001e-05 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683842+0000", 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756969+0000", 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073126925999999995 2024-04-24T00:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757172+0000", 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020269900000000001 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757708+0000", 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053609999999999997 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127561 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 251904~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.425977058999997, 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074620636000000004, 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127561 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683866+0000", 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.127e-06 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683873+0000", 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683893+0000", 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9300000000000002e-05 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683954+0000", 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1148999999999993e-05 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757783+0000", 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073829025000000006 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757939+0000", 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582700000000001 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758486+0000", 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054703200000000003 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127562 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 256000~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.425855464000001, 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075112683, 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127562 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683988+0000", 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.443e-06 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684000+0000", 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.252e-05 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684025+0000", 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4956e-05 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684044+0000", 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8770000000000002e-05 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758566+0000", 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074522094999999997 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758669+0000", 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010247399999999999 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759099+0000", 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043042500000000001 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127563 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 260096~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.425761631999997, 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075791184999999997, 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127563 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684082+0000", 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684089+0000", 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4889999999999998e-06 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684115+0000", 2024-04-24T00:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5643000000000002e-05 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684133+0000", 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7842999999999998e-05 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759174+0000", 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075041103999999997 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759329+0000", 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015504200000000001 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759872+0000", 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054272399999999996 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:36.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:36.203 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:36.198+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:36.304 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:36.298+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:36.323 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:36.323 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:36.323 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:36.323 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:36.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:36.425 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:36.422+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:36.506 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:36.502+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:36.508 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:36.508 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:36.508 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:36.508 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:36.508 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:36.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:36.595 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:37:36.607 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:36.602+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 203.48531932099999, 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:37:36.700 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 203.48496840600001, 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:37:36.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:37:36.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:37:36.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 203.47808008600001, 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:36.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:37:36.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.21979845999999, 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.20952204400001, 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.747571241, 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.72153722, 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.507594313, 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.497582369, 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.739 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.224203507, 2024-04-24T00:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.207535058, 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.021637275, 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.337878918000001, 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.315661919999997, 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 81.011092082000005, 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.998467121999994, 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.648975634999999, 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.63395744, 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.741141037, 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.703789009999999, 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:36.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:36.799 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:36.794+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:36.899 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:36.894+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:36.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:36.943 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:36.943 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:36.943 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:36.943 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:36.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:37.000 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:36.994+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:37.080 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:37.080 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:37.080 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:37.080 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:37.080 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:37.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:37.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:37.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:37.100 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:37.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:37.100 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:37.094+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:37.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:37.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:37.135 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:37.135 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:37.153 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:37.153 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:37.173 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:37.174 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:37.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:37.195 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:37.201 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:37.198+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:37.213 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:37.213 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:37.217 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:37.217 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:37.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:37.302 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:37.298+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:37.391 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:37.391 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:37.404 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:37.402+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:37.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:37.505 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:37.502+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:37.528 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:37.528 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:37.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:37.606 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:37.602+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:37.707 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:37.702+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:37.725 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:37.725 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:37.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:37.808 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:37.802+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:37.909 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:37.906+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:37.910 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:37.910 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:37.924 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:38.013 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:38.010+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:38.093 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:38.093 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:38.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:38.110 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:38.106+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:38.211 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:38.206+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:38.272 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:38.273 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:38.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:38.332 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:38.330+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:38.405 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:38.405 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:38.413 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:38.406+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:38.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:38.513 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:38.510+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:38.584 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:38.585 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:38.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:38.615 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:38.610+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:38.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:37:38.716 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:38.710+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:38.775 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:38.776 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:38.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:38.841 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:38.838+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:38.909 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:38.910 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:38.917 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:38.914+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:38.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:39.018 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:39.014+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:39.091 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:39.091 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:39.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:39.149 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:39.146+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:39.221 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:39.214+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:39.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:39.277 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:39.278 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:39.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:39.341 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:39.338+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:39.380 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:39.381 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:39.397 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:39.397 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:39.414 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:39.414 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:39.422 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:39.418+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:39.430 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:39.431 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:39.451 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:39.451 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:39.463 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:39.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:39.463 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:39.463 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:39.472 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:39.472 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:39.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:39.493 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:39.493 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:39.514 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:39.515 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:39.523 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:39.518+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:39.593 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:37:39.623 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:39.618+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:39.631 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:39.631 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:39.631 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:39.631 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:39.631 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:39.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:39.725 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:39.722+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:39.826 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:39.822+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:39.830 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:39.830 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 192.39192781400001, 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.831 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:39.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:37:39.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 192.33641632800001, 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 173.23583968, 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:37:39.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:37:39.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 173.194274726, 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:37:39.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.63054316399999, 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:37:39.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:37:39.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:37:39.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 132.12756976700001, 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.858 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:37:39.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:37:39.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:37:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.889255534, 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:39.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.85431511100001, 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:37:39.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:37:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.85429423799999, 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.873 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.877 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.854270859, 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.881 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.854249317, 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:37:39.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.85422779699999, 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.886 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.85418163999999, 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:37:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:37:39.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.854157521, 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.895 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:37:39.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:37:39.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.854134117, 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:37:39.899 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:37:39.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.854106344, 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:39.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:37:39.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:37:39.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.854084931, 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:37:39.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:37:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.85406433999999, 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:37:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.63799409799999, 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:37:39.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:37:39.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:37:39.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.61002319400001, 2024-04-24T00:37:39.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:39.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:37:39.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:37:39.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:39.926 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:39.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:39.928 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:39.922+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:40.028 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:40.022+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:40.102 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:40.102 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:40.102 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:40.102 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:40.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:40.129 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:40.126+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:40.231 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:40.230+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:40.284 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:40.284 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:40.284 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:40.284 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:40.284 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:40.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:40.345 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:40.342+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:40.421 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.176335109999997, 2024-04-24T00:37:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:37:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:37:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:37:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:37:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:37:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:37:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:37:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:37:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:37:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:37:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:37:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:37:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:37:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:37:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:37:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:37:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:37:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:37:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:37:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:37:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.146567629000003, 2024-04-24T00:37:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:37:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:37:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:37:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:37:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:37:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:37:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:37:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:37:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:37:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:37:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:37:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:37:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:37:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126791 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 112640~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.777002600000003, 2024-04-24T00:37:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074437884999999995, 2024-04-24T00:37:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126791 2024-04-24T00:37:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:37:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640034+0000", 2024-04-24T00:37:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-04-24T00:37:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640044+0000", 2024-04-24T00:37:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2660000000000007e-06 2024-04-24T00:37:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640050+0000", 2024-04-24T00:37:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713112+0000", 2024-04-24T00:37:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073062031999999999 2024-04-24T00:37:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713345+0000", 2024-04-24T00:37:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000232616 2024-04-24T00:37:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714469+0000", 2024-04-24T00:37:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011237669999999999 2024-04-24T00:37:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126792 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 116736~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.776973826999999, 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075734135999999994, 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126792 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3399999999999999e-07 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640063+0000", 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7e-06 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640072+0000", 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0729999999999994e-06 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640084+0000", 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2486e-05 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714588+0000", 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074503102000000002 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714763+0000", 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175885 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715794+0000", 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001030256 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126793 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 120832~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.776940037000003, 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076915124000000001, 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126793 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640094+0000", 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640097+0000", 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9799999999999998e-06 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640107+0000", 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6229999999999992e-06 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640123+0000", 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5892999999999998e-05 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715884+0000", 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075761384000000001 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.716061+0000", 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176727 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717009+0000", 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094787400000000003 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126794 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 124928~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.776901358000003, 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078057396000000001, 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126794 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640133+0000", 2024-04-24T00:37:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-07 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640136+0000", 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1e-06 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640145+0000", 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.516e-06 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640152+0000", 2024-04-24T00:37:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717088+0000", 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076936302999999998 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717201+0000", 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112704 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718189+0000", 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098839299999999995 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126795 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 129024~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.776872472000001, 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079207986999999994, 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126795 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640162+0000", 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7400000000000003e-07 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640165+0000", 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1860000000000001e-06 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640174+0000", 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5429999999999999e-06 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640181+0000", 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8029999999999999e-06 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718272+0000", 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078091277000000001 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718443+0000", 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171027 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.719369+0000", 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092547699999999996 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127300 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 229376~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.754616212999998, 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073789773000000003, 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127300 2024-04-24T00:37:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662418+0000", 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.203e-06 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662439+0000", 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0713e-05 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662460+0000", 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.105e-05 2024-04-24T00:37:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662488+0000", 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7966e-05 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735368+0000", 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072880167999999995 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735555+0000", 2024-04-24T00:37:40.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018692399999999999 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736207+0000", 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065174899999999997 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127301 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 233472~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.754515253999998, 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074400210999999994, 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127301 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662520+0000", 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8700000000000001e-06 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662527+0000", 2024-04-24T00:37:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7830000000000001e-06 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662549+0000", 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2379e-05 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662564+0000", 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.473e-05 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736281+0000", 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073717486999999998 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736386+0000", 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104594 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736918+0000", 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053236799999999997 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127302 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.754447329000001, 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075297446000000004, 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127302 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662587+0000", 2024-04-24T00:37:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096e-06 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662608+0000", 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662624+0000", 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5563e-05 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662639+0000", 2024-04-24T00:37:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4766999999999999e-05 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737013+0000", 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074374061000000005 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737195+0000", 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182201 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737884+0000", 2024-04-24T00:37:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068866800000000003 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127303 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.754373450999999, 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075989524000000003, 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127303 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662662+0000", 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5400000000000001e-06 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662668+0000", 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1970000000000001e-06 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662689+0000", 2024-04-24T00:37:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0953000000000001e-05 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662703+0000", 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737962+0000", 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075258932000000001 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738074+0000", 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011173800000000001 2024-04-24T00:37:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738649+0000", 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057545399999999998 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127304 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.754309118000002, 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076894431999999999, 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127304 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662725+0000", 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0359999999999999e-06 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662732+0000", 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6440000000000003e-06 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662751+0000", 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8583000000000001e-05 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662768+0000", 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7336000000000001e-05 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738744+0000", 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075976005999999999 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738924+0000", 2024-04-24T00:37:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017996900000000001 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739619+0000", 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069485800000000002 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127305 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.754244843000002, 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077646889999999996, 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127305 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662790+0000", 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662796+0000", 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3659999999999997e-06 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662819+0000", 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2503000000000001e-05 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662834+0000", 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739755+0000", 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076920905999999997 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739876+0000", 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121445 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740435+0000", 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055911299999999997 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127306 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.75417728, 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078485103000000001, 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127306 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662857+0000", 2024-04-24T00:37:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1599999999999999e-06 2024-04-24T00:37:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662864+0000", 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3829999999999997e-06 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662883+0000", 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9476000000000001e-05 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662897+0000", 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3485999999999999e-05 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740564+0000", 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077667768999999998 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740736+0000", 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017116700000000001 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741341+0000", 2024-04-24T00:37:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060566199999999998 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127307 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.754117874999999, 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079342509000000006, 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127307 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662917+0000", 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662922+0000", 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662939+0000", 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833e-05 2024-04-24T00:37:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662954+0000", 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741518+0000", 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078564118000000002 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741709+0000", 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191337 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.742258+0000", 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054904499999999996 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127559 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 243712~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.733333569999999, 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073184308000000003, 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127559 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683701+0000", 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2300000000000001e-06 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683724+0000", 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.321e-05 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683745+0000", 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.101e-05 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683761+0000", 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756124+0000", 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072363532999999994 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756327+0000", 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020239999999999999 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756884+0000", 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055760300000000005 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127560 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 247808~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.733249831000002, 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073924043999999994, 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127560 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683785+0000", 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1829999999999999e-06 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683792+0000", 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2799999999999998e-06 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683810+0000", 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8076000000000001e-05 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683842+0000", 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756969+0000", 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073126925999999995 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757172+0000", 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020269900000000001 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757708+0000", 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053609999999999997 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127561 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 251904~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.733168503000002, 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074620636000000004, 2024-04-24T00:37:40.512 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127561 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683866+0000", 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.127e-06 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683873+0000", 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:37:40.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683893+0000", 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9300000000000002e-05 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683954+0000", 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1148999999999993e-05 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757783+0000", 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073829025000000006 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757939+0000", 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582700000000001 2024-04-24T00:37:40.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758486+0000", 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054703200000000003 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127562 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 256000~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.733046907999999, 2024-04-24T00:37:40.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075112683, 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127562 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683988+0000", 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.443e-06 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684000+0000", 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.252e-05 2024-04-24T00:37:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684025+0000", 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4956e-05 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684044+0000", 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8770000000000002e-05 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758566+0000", 2024-04-24T00:37:40.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074522094999999997 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758669+0000", 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010247399999999999 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759099+0000", 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043042500000000001 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127563 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 260096~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.732953076000001, 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075791184999999997, 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127563 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.522 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684082+0000", 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:37:40.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684089+0000", 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4889999999999998e-06 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684115+0000", 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5643000000000002e-05 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684133+0000", 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7842999999999998e-05 2024-04-24T00:37:40.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759174+0000", 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075041103999999997 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759329+0000", 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015504200000000001 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759872+0000", 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054272399999999996 2024-04-24T00:37:40.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:40.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:40.526 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:40.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:40.528 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:40.522+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:40.628 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:40.622+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:40.648 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5] out_osds: [7, 0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:40.648 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:40.648 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-24T00:37:40.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-04-24T00:37:40.712 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:40.712 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:40.712 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:40.712 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:40.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:40.730 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:40.726+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:40.830 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:40.826+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:40.894 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:40.895 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:40.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:40.953 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:40.950+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:40.970 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:37:40.966+0000 7f841a5f8640 -1 mon.a@0(leader).osd e228 definitely_dead 0 2024-04-24T00:37:41.032 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:41.030+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:41.088 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 207.871610309, 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:41.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:37:41.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:37:41.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:41.094 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 207.87125939399999, 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:41.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:37:41.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:37:41.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:37:41.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 207.86437107399999, 2024-04-24T00:37:41.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:41.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:37:41.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.60608944800001, 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.110 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.595813032, 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:37:41.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.114 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.133862229, 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:41.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:37:41.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:37:41.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 127.107828208, 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:37:41.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 126.893885301, 2024-04-24T00:37:41.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:41.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:37:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:37:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:37:41.129 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 126.883873357, 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:37:41.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.610494495, 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.134 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:41.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:37:41.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 117.593826046, 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:37:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 112.407928263, 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.144 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:37:41.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:37:41.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:37:41.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.150 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.724169906, 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:37:41.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:37:41.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.156 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.701952907999996, 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:41.161 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.397383070000004, 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:37:41.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:37:41.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.384758110000007, 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:41.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.035266622999998, 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:37:41.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.020248428000002, 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:37:41.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:37:41.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.127432025000001, 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:37:41.182 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:37:41.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:37:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.090079998, 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:41.192 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:41.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:41.195 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:41.190+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:41.296 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:41.290+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:41.326 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:41.326 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:41.326 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:41.326 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:41.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:41.396 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:41.390+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:41.459 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:41.459 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:41.459 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:41.459 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:41.459 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:41.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:41.517 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:41.514+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:41.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:41.598 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:41.594+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:41.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:37:41.630 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.0. 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 208.43398351499999, 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.657 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 194.012197821, 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:41.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.870070255, 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:37:41.663 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:37:41.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.669 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.732298473, 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.670 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:41.675 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.694816155, 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:37:41.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:37:41.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:37:41.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.680 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 124.032470291, 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 124.022612255, 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.686 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:37:41.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:41.691 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 124.00226264699999, 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:41.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:37:41.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:37:41.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:37:41.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:37:41.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.96821731199999, 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.698 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:37:41.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.701 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.702 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.965907476, 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:37:41.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:37:41.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.018189584, 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.708 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:41.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:41.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:41.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 96.128799588000007, 2024-04-24T00:37:41.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:37:41.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:37:41.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 96.085323790999993, 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.723 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 91.111350489000003, 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.724 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:37:41.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:37:41.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 82.132921033000002, 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.730 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:37:41.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:37:41.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 82.098918611000002, 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:37:41.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157093 3.13 3:cd0b7234:::rbd_data.115ee709dad1.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.762313374999998, 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12167272799999999, 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.740 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157093 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:41.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885996+0000", 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.426e-06 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886003+0000", 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8859999999999998e-06 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886020+0000", 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6135000000000001e-05 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886027+0000", 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3379999999999999e-06 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886099+0000", 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1802000000000001e-05 2024-04-24T00:37:41.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886260+0000", 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016156800000000001 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.895327+0000", 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090669270000000007 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007496+0000", 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112168567 2024-04-24T00:37:41.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007520+0000", 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4646000000000001e-05 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007599+0000", 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8424999999999995e-05 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007668+0000", 2024-04-24T00:37:41.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9008000000000001e-05 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:157093 3.13 e197/106 ondisk, result = 0)", 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 34.730895337, 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090093925000000005, 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:41.746 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917416+0000", 2024-04-24T00:37:41.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917426+0000", 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917465+0000", 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9295999999999998e-05 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917484+0000", 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007420+0000", 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089936099000000005 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007435+0000", 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4966999999999999e-05 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007507+0000", 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2206000000000006e-05 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169765 3.13 3:c9e19d2e:::rbd_data.115ee24d4bb.0000000000000011:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.788529737000001, 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096474246999999999, 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:41.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169765 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:41.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859781+0000", 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859792+0000", 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859819+0000", 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7025999999999998e-05 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859833+0000", 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3763e-05 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859956+0000", 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122989 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.860147+0000", 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190843 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.866555+0000", 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064084540000000001 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.955923+0000", 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089367567999999994 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956056+0000", 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013292600000000001 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956157+0000", 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101029 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956253+0000", 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5921999999999994e-05 2024-04-24T00:37:41.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169765 3.13 e210/106 ondisk, result = 0)", 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.769248303000001, 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077208636999999997, 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:41.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879062+0000", 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879072+0000", 2024-04-24T00:37:41.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0149999999999999e-05 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879106+0000", 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3841999999999999e-05 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879126+0000", 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9264000000000001e-05 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:41.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956025+0000", 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076899281999999999 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956045+0000", 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9902999999999999e-05 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956269+0000", 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000224012 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.759 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:41.760 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:41.760 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:41.762 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-24T00:37:41.762 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:41.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:41.763 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:41.758+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:41.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:41.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:41.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:41.764 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:41.764 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:41.766 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:41.766 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:41.802 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:41.802 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:41.821 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:41.821 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:41.841 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:41.841 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:41.860 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:41.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:41.861 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:41.858+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:41.937 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:41.937 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:41.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:41.962 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:41.958+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:42.063 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:42.058+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:42.072 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:42.072 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:42.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:42.164 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:42.158+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:42.210 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:42.210 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:42.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:42.265 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:42.262+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:42.366 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:42.362+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:42.392 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:42.392 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:42.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:42.467 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:42.462+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:42.568 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:42.566+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:42.571 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:42.571 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:42.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:42.635 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:37:42.709 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:42.706+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:42.749 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:42.749 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:42.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:42.808 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:42.806+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:42.871 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:42.866+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:42.886 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:42.886 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:42.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:42.997 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:42.994+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:43.070 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:43.070 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:43.074 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:43.070+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:43.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:43.175 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:43.170+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:43.254 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:43.254 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:43.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:43.313 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:43.310+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:43.376 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:43.374+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:43.405 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:43.405 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:43.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:43.505 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:43.502+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:43.578 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:43.574+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:43.586 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:43.586 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:43.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:43.679 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:43.674+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:43.780 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:43.774+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:43.801 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:43.801 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:43.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:43.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:43.909 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:43.906+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:43.983 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:43.978+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:44.009 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:44.009 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:44.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:44.071 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:44.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:44.084 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:44.082+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:44.091 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:44.091 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:44.116 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:44.116 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:44.135 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:44.135 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:44.161 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:44.161 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:44.174 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:44.175 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:44.175 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:44.175 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:44.175 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:44.184 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:44.184 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:44.185 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:44.178+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:44.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:44.207 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:44.208 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:44.230 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:44.297 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:44.294+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:21644 3.4 3:21dccd79:::rbd_data.115eeb9272a5.0000000000000036:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 196.89442455599999, 2024-04-24T00:37:44.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197854482, 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 21644 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433991+0000", 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.433994+0000", 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2760000000000001e-06 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434009+0000", 2024-04-24T00:37:44.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434046+0000", 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6251000000000002e-05 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434068+0000", 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2855999999999999e-05 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434193+0000", 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012482500000000001 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.434338+0000", 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014447799999999999 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.448510+0000", 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014172644 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631587+0000", 2024-04-24T00:37:44.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18307663800000001 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631622+0000", 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4535000000000001e-05 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631713+0000", 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1949999999999999e-05 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631846+0000", 2024-04-24T00:37:44.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132225 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:21644 3.4 e52/21 ondisk, result = 0)", 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 196.83891306999999, 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14210198099999999, 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489503+0000", 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489505+0000", 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489514+0000", 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1120000000000002e-06 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489546+0000", 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1782999999999997e-05 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.489562+0000", 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5872e-05 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631560+0000", 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14199825699999999 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631576+0000", 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6308999999999999e-05 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:27.631605+0000", 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8425e-05 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:33746 3.6 3:605badee:::rbd_data.115ec4f9b3a7.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.738336422, 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.149234539, 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33746 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.341 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590079+0000", 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590081+0000", 2024-04-24T00:37:44.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9800000000000001e-06 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590092+0000", 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0798999999999999e-05 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590124+0000", 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-05 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590139+0000", 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5373e-05 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590254+0000", 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114827 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.590385+0000", 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013166 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.599189+0000", 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088031700000000008 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739160+0000", 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13997095600000001 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:44.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739201+0000", 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1699999999999997e-05 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739266+0000", 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4428e-05 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739314+0000", 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8155999999999998e-05 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:33746 3.6 e71/59 ondisk, result = 0)", 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.69677146800001, 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107678365, 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631644+0000", 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631645+0000", 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7399999999999998e-07 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631653+0000", 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9019999999999999e-06 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631717+0000", 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4243000000000001e-05 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.631740+0000", 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2872000000000001e-05 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739185+0000", 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10744498 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739195+0000", 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0577999999999999e-05 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:46.739323+0000", 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012731600000000001 2024-04-24T00:37:44.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:77659 3.6 3:63fac15a:::rbd_data.115e805f80b3.0000000000000019:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.13303990599999, 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104931643, 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77659 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.354 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195376+0000", 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195378+0000", 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0480000000000001e-06 2024-04-24T00:37:44.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195388+0000", 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0426e-05 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195413+0000", 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4185e-05 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195428+0000", 2024-04-24T00:37:44.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5058999999999999e-05 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195581+0000", 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015295900000000001 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.195802+0000", 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022136000000000001 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.206240+0000", 2024-04-24T00:37:44.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010438159000000001 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299932+0000", 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093691568000000003 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.299988+0000", 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6567999999999998e-05 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300156+0000", 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167305 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:24.300308+0000", 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015200599999999999 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:78059 3.12 3:4e98e324:::rbd_data.115e805f80b3.000000000000000d:head [write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:44.359 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.63006650899999, 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0227916699999999, 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 78059 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:44.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698349+0000", 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698353+0000", 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2540000000000001e-06 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698368+0000", 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5705e-05 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698394+0000", 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6208e-05 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698407+0000", 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2818e-05 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698727+0000", 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031990399999999998 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.698736+0000", 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9090000000000006e-06 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703834+0000", 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005098132 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:27.703845+0000", 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0966e-05 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721053+0000", 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0172082490000001 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:28.721141+0000", 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7525e-05 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.364 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81501 3.e e111/107 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head v 111'228, mlcod=108'226)", 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.39175227600001, 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095939886000000002, 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.936664+0000", 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1800000000000005e-07 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944273+0000", 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076088070000000004 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944290+0000", 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944297+0000", 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4050000000000003e-06 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944354+0000", 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6919000000000002e-05 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.944370+0000", 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5730999999999999e-05 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032529+0000", 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088159159000000001 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.032603+0000", 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4371999999999999e-05 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81622 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1261568~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.35681185300001, 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079717599, 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81622 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971604+0000", 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3500000000000002e-07 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971606+0000", 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0619999999999999e-06 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971613+0000", 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1979999999999998e-06 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971618+0000", 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4639999999999997e-06 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051210+0000", 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079591890999999998 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051258+0000", 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7876e-05 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051322+0000", 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3672999999999998e-05 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81623 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1409024~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.373 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.35679098, 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079956736, 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81623 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971625+0000", 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971628+0000", 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.452e-06 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971634+0000", 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000001e-06 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971640+0000", 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5140000000000001e-06 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051362+0000", 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079722017000000006 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051444+0000", 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2532000000000002e-05 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051582+0000", 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137176 2024-04-24T00:37:44.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81624 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1556480~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.356767601, 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080148567000000004, 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81624 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971648+0000", 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971649+0000", 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1399999999999997e-07 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971651+0000", 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2469999999999998e-06 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971657+0000", 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4919999999999998e-06 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971663+0000", 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2299999999999999e-06 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051618+0000", 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079955766999999997 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051696+0000", 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7665999999999998e-05 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051797+0000", 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100751 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81625 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1835008~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.35674605899999, 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080457545000000005, 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81625 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971670+0000", 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600000000000002e-07 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971672+0000", 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.153e-06 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971679+0000", 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4099999999999996e-06 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971684+0000", 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1390000000000003e-06 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.051988+0000", 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080304269999999997 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052038+0000", 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0096999999999998e-05 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052127+0000", 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9060000000000005e-05 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81626 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 1982464~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:44.386 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.356724539, 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080574955000000004, 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81626 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971691+0000", 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971692+0000", 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5799999999999999e-07 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971694+0000", 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5119999999999998e-06 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971702+0000", 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8830000000000006e-06 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971728+0000", 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5681e-05 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052165+0000", 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080437243000000005 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052206+0000", 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0858000000000001e-05 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052266+0000", 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.012e-05 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81627 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2129920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.35667838200001, 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080660265999999994, 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81627 2024-04-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971737+0000", 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971738+0000", 2024-04-24T00:37:44.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4000000000000001e-07 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971741+0000", 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0319999999999999e-06 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971748+0000", 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3139999999999998e-06 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971754+0000", 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5400000000000003e-06 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052299+0000", 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080545145999999998 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052339+0000", 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-05 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052398+0000", 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8884000000000001e-05 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81628 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2277376~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.356654263, 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080762269999999997, 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81628 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971762+0000", 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971765+0000", 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4719999999999998e-06 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971771+0000", 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.302e-06 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971777+0000", 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3040000000000001e-06 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052430+0000", 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080652492000000006 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052469+0000", 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9728000000000003e-05 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052524+0000", 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4416000000000002e-05 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81629 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2555904~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.35663085900001, 2024-04-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080975944999999994, 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81629 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971785+0000", 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971792+0000", 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5690000000000001e-06 2024-04-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971798+0000", 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2559999999999999e-06 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971803+0000", 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2290000000000002e-06 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052572+0000", 2024-04-24T00:37:44.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080768908 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052648+0000", 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5833000000000004e-05 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052761+0000", 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112685 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81630 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2703360~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.35660308600001, 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081951908000000004, 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81630 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971813+0000", 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:37:44.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971815+0000", 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2589999999999999e-06 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971822+0000", 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3049999999999998e-06 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971827+0000", 2024-04-24T00:37:44.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5509999999999999e-06 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052788+0000", 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080960937999999996 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.052873+0000", 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4358999999999996e-05 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053765+0000", 2024-04-24T00:37:44.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089205399999999998 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81631 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2850816~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.35658167299999, 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082168460999999998, 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.408 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81631 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971834+0000", 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971835+0000", 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2399999999999999e-07 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971837+0000", 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1220000000000002e-06 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971843+0000", 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2339999999999999e-06 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971848+0000", 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1919999999999996e-06 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053843+0000", 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081994782000000002 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.053914+0000", 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1278000000000004e-05 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054003+0000", 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8429000000000003e-05 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81632 3.6 3:632280d1:::rbd_data.115e805f80b3.0000000000000006:2a [read 2998272~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.35656108200001, 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082367669000000004, 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81632 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971855+0000", 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971857+0000", 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1330000000000002e-06 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971863+0000", 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9889999999999997e-06 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.971869+0000", 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-06 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054047+0000", 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082178560999999997 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054143+0000", 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6020000000000006e-05 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.054222+0000", 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9220000000000004e-05 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:87860 3.4 3:224743f6:::rbd_data.115e12b71652.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-04-24T00:37:44.416 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.14049084, 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12763875, 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87860 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187925+0000", 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187928+0000", 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6869999999999999e-06 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187938+0000", 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0543000000000001e-05 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187966+0000", 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7670000000000001e-05 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.187985+0000", 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9522e-05 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188056+0000", 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0658e-05 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.188116+0000", 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9812999999999998e-05 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.194183+0000", 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060670769999999997 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315204+0000", 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.121021006 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315330+0000", 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012618199999999999 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315429+0000", 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8667999999999998e-05 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315564+0000", 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000134924 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:87860 3.4 e116/21 ondisk, result = 0)", 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.112519936, 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099686362000000001, 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215896+0000", 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215898+0000", 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.12e-06 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215907+0000", 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7099999999999996e-06 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215939+0000", 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2583000000000003e-05 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.215956+0000", 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6595999999999998e-05 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315301+0000", 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099344996000000005 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315319+0000", 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8559999999999998e-05 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:39.315582+0000", 2024-04-24T00:37:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000262797 2024-04-24T00:37:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.426 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:44.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:44.428 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:44.422+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:44.528 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:44.526+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:44.620 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:44.620 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:44.629 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:44.626+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:44.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:44.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:37:44.731 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:44.726+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:44.778 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:44.778 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:44.778 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:44.778 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:44.778 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:44.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:44.832 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:44.826+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:44.925 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:44.925 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.682422575999993, 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.926 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:37:44.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:37:44.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.652655095, 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126791 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 112640~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.935 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.283090066, 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074437884999999995, 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126791 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640034+0000", 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640044+0000", 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2660000000000007e-06 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640050+0000", 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:44.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713112+0000", 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073062031999999999 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713345+0000", 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000232616 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714469+0000", 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011237669999999999 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126792 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 116736~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.283061293000003, 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075734135999999994, 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126792 2024-04-24T00:37:44.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:44.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3399999999999999e-07 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640063+0000", 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7e-06 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640072+0000", 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0729999999999994e-06 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640084+0000", 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2486e-05 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714588+0000", 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074503102000000002 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714763+0000", 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175885 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715794+0000", 2024-04-24T00:37:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001030256 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126793 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 120832~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.283027503, 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076915124000000001, 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126793 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640094+0000", 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640097+0000", 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9799999999999998e-06 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640107+0000", 2024-04-24T00:37:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6229999999999992e-06 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640123+0000", 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5892999999999998e-05 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715884+0000", 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075761384000000001 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.716061+0000", 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176727 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717009+0000", 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094787400000000003 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126794 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 124928~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.282988824, 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078057396000000001, 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126794 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640133+0000", 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-07 2024-04-24T00:37:44.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640136+0000", 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1e-06 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640145+0000", 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.516e-06 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640152+0000", 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:44.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717088+0000", 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076936302999999998 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717201+0000", 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112704 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718189+0000", 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098839299999999995 2024-04-24T00:37:44.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126795 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 129024~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.282959937999998, 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079207986999999994, 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126795 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640162+0000", 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7400000000000003e-07 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640165+0000", 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1860000000000001e-06 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640174+0000", 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5429999999999999e-06 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640181+0000", 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8029999999999999e-06 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718272+0000", 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078091277000000001 2024-04-24T00:37:44.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718443+0000", 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171027 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.719369+0000", 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092547699999999996 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.957 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127300 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 229376~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.260703679000002, 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073789773000000003, 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127300 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662418+0000", 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.203e-06 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662439+0000", 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0713e-05 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662460+0000", 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.105e-05 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662488+0000", 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7966e-05 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735368+0000", 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072880167999999995 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735555+0000", 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018692399999999999 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736207+0000", 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065174899999999997 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.961 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127301 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 233472~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.260602720000001, 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074400210999999994, 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127301 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662520+0000", 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8700000000000001e-06 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662527+0000", 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7830000000000001e-06 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662549+0000", 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2379e-05 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662564+0000", 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.473e-05 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736281+0000", 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073717486999999998 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736386+0000", 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104594 2024-04-24T00:37:44.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736918+0000", 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053236799999999997 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127302 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.260534794999998, 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075297446000000004, 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.966 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127302 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662587+0000", 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096e-06 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662608+0000", 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-04-24T00:37:44.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662624+0000", 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5563e-05 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662639+0000", 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4766999999999999e-05 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737013+0000", 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074374061000000005 2024-04-24T00:37:44.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737195+0000", 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182201 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737884+0000", 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068866800000000003 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127303 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.260460917000003, 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075989524000000003, 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127303 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662662+0000", 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5400000000000001e-06 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662668+0000", 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1970000000000001e-06 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662689+0000", 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0953000000000001e-05 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662703+0000", 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737962+0000", 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075258932000000001 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738074+0000", 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011173800000000001 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738649+0000", 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057545399999999998 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127304 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.260396583999999, 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076894431999999999, 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127304 2024-04-24T00:37:44.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662725+0000", 2024-04-24T00:37:44.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0359999999999999e-06 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662732+0000", 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6440000000000003e-06 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662751+0000", 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8583000000000001e-05 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662768+0000", 2024-04-24T00:37:44.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7336000000000001e-05 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738744+0000", 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075976005999999999 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738924+0000", 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017996900000000001 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739619+0000", 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069485800000000002 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127305 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.260332308999999, 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077646889999999996, 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127305 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:44.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662790+0000", 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662796+0000", 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3659999999999997e-06 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662819+0000", 2024-04-24T00:37:44.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2503000000000001e-05 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662834+0000", 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739755+0000", 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076920905999999997 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739876+0000", 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121445 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740435+0000", 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055911299999999997 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127306 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:44.983 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.260264745999997, 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078485103000000001, 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127306 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662857+0000", 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1599999999999999e-06 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662864+0000", 2024-04-24T00:37:44.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3829999999999997e-06 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662883+0000", 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9476000000000001e-05 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662897+0000", 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3485999999999999e-05 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740564+0000", 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077667768999999998 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740736+0000", 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017116700000000001 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741341+0000", 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060566199999999998 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127307 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.260205341000002, 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079342509000000006, 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127307 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662917+0000", 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662922+0000", 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662939+0000", 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833e-05 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662954+0000", 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:44.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741518+0000", 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078564118000000002 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741709+0000", 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191337 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.742258+0000", 2024-04-24T00:37:44.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054904499999999996 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127559 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 243712~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.239421036000003, 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073184308000000003, 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127559 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683701+0000", 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2300000000000001e-06 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683724+0000", 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.321e-05 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683745+0000", 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.101e-05 2024-04-24T00:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683761+0000", 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756124+0000", 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072363532999999994 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756327+0000", 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020239999999999999 2024-04-24T00:37:44.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756884+0000", 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055760300000000005 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127560 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 247808~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.239337296999999, 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073924043999999994, 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:44.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127560 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:44.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683785+0000", 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1829999999999999e-06 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683792+0000", 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2799999999999998e-06 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683810+0000", 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8076000000000001e-05 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683842+0000", 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756969+0000", 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073126925999999995 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:44.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757172+0000", 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020269900000000001 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757708+0000", 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053609999999999997 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.000 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127561 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 251904~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.239255968999998, 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074620636000000004, 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127561 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683866+0000", 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.127e-06 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683873+0000", 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683893+0000", 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9300000000000002e-05 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683954+0000", 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1148999999999993e-05 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757783+0000", 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073829025000000006 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757939+0000", 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582700000000001 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758486+0000", 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054703200000000003 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127562 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 256000~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.239134374000002, 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075112683, 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127562 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683988+0000", 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.443e-06 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684000+0000", 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.252e-05 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684025+0000", 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4956e-05 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684044+0000", 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8770000000000002e-05 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758566+0000", 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074522094999999997 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758669+0000", 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010247399999999999 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759099+0000", 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043042500000000001 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127563 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 260096~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.239040541999998, 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075791184999999997, 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127563 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684082+0000", 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684089+0000", 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4889999999999998e-06 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684115+0000", 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5643000000000002e-05 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684133+0000", 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7842999999999998e-05 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759174+0000", 2024-04-24T00:37:45.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075041103999999997 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759329+0000", 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015504200000000001 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759872+0000", 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054272399999999996 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.013 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.014 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:45.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:45.016 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:45.010+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:45.116 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:45.110+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGRecoveryDeleteReply(3.17 e232,230 3:ee09fe34:::rbd_data.115e397a9cd4.000000000000011a:head,221'309)", 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:45.148286+0000", 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00058879699999999995, 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060530200000000005, 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.150 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.148286+0000", 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.148286+0000", 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.148287+0000", 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860000000000001e-06 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.148295+0000", 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9500000000000001e-06 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.148319+0000", 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3940000000000001e-05 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.148333+0000", 2024-04-24T00:37:45.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3475999999999999e-05 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.148423+0000", 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0632999999999998e-05 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-04-24T00:37:45.153 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:45.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:45.217 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:45.214+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:45.318 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:45.314+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:45.346 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:45.346 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:45.347 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:45.347 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:45.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:45.419 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:45.414+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:45.512 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:45.512 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.29621665900001, 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:37:45.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.295865744, 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:37:45.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:37:45.524 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.288977424, 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 161.03069579800001, 2024-04-24T00:37:45.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:37:45.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:37:45.533 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 161.020419382, 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:37:45.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 131.55846857899999, 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:45.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:37:45.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 131.53243455800001, 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 131.31849165099999, 2024-04-24T00:37:45.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:37:45.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 131.308479707, 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:37:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:37:45.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.035100845, 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:37:45.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:37:45.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.01843239599999, 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.560 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:37:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.83253461300001, 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:45.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:37:45.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:37:45.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:45.570 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.148776256, 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.571 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:45.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:37:45.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:37:45.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:45.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 103.126559258, 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 89.821989419999994, 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.580 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:37:45.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:37:45.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:37:45.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 89.809364459999998, 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:37:45.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:37:45.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:37:45.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.459872973000003, 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.591 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:37:45.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.591 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:45.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:45.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:37:45.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.444854778, 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:45.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:37:45.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:37:45.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.601 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.552038374999999, 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:45.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.514686348000001, 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:45.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:45.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:45.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:45.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:45.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:37:45.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:37:45.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.611 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:45.611 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:45.611 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:45.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:45.613 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:37:45.613 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:45.610+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:45.713 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:45.710+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:45.754 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:45.754 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:45.754 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:45.754 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:45.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:45.814 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:45.810+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:45.885 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:45.886 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:45.886 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:45.886 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:45.886 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:45.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:45.948 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:45.946+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:46.015 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:46.010+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 212.80902349300001, 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.027 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:46.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:37:46.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 198.38723779899999, 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.033 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:37:46.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:37:46.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:37:46.037 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 137.24511023299999, 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.038 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:37:46.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:46.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.107338451, 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.069856133, 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.050 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:37:46.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 128.407510269, 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.057 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:46.062 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 128.397652233, 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:37:46.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:37:46.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:46.068 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 128.377302625, 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:37:46.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.070 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:46.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:46.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:37:46.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:37:46.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:37:46.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:37:46.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:37:46.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:37:46.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:37:46.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:37:46.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:37:46.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:37:46.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:46.077 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 117.34325729, 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 117.340947454, 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:37:46.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:37:46.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:46.088 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.393229562, 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:37:46.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:37:46.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:37:46.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:37:46.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.503839566, 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.460363769, 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.100 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.486390467000007, 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.104 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:37:46.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:37:46.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.507961011000006, 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.110 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:37:46.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:37:46.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.473958589000006, 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157093 3.13 3:cd0b7234:::rbd_data.115ee709dad1.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:46.119 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 39.137353353000002, 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12167272799999999, 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157093 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:46.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885996+0000", 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.426e-06 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886003+0000", 2024-04-24T00:37:46.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8859999999999998e-06 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886020+0000", 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6135000000000001e-05 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886027+0000", 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3379999999999999e-06 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886099+0000", 2024-04-24T00:37:46.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1802000000000001e-05 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886260+0000", 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016156800000000001 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.895327+0000", 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090669270000000007 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007496+0000", 2024-04-24T00:37:46.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112168567 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007520+0000", 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4646000000000001e-05 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007599+0000", 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8424999999999995e-05 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007668+0000", 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9008000000000001e-05 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:157093 3.13 e197/106 ondisk, result = 0)", 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 39.105935315000004, 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090093925000000005, 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:46.125 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917416+0000", 2024-04-24T00:37:46.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917426+0000", 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917465+0000", 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9295999999999998e-05 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917484+0000", 2024-04-24T00:37:46.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007420+0000", 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089936099000000005 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007435+0000", 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4966999999999999e-05 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007507+0000", 2024-04-24T00:37:46.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2206000000000006e-05 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169765 3.13 3:c9e19d2e:::rbd_data.115ee24d4bb.0000000000000011:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.163569715000001, 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096474246999999999, 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:46.129 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169765 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859781+0000", 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859792+0000", 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859819+0000", 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7025999999999998e-05 2024-04-24T00:37:46.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859833+0000", 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3763e-05 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859956+0000", 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122989 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.860147+0000", 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190843 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.866555+0000", 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064084540000000001 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.955923+0000", 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089367567999999994 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956056+0000", 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013292600000000001 2024-04-24T00:37:46.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956157+0000", 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101029 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956253+0000", 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5921999999999994e-05 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169765 3.13 e210/106 ondisk, result = 0)", 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 24.144288281000001, 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077208636999999997, 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879062+0000", 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879072+0000", 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0149999999999999e-05 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879106+0000", 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3841999999999999e-05 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879126+0000", 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9264000000000001e-05 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956025+0000", 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076899281999999999 2024-04-24T00:37:46.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956045+0000", 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9902999999999999e-05 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956269+0000", 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000224012 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:46.138 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:46.139 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:46.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:46.141 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:46.134+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:46.241 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:46.234+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:46.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:46.311 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:46.311 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:46.311 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:46.311 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:46.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:46.341 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:46.338+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:46.420 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:46.421 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:46.438 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:46.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:46.442 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:46.438+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:46.450 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:46.450 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:46.450 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:46.450 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:46.450 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:46.456 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:46.456 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:46.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:46.475 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:46.475 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:46.495 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:46.495 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:46.513 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:46.513 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:46.530 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:46.543 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:46.538+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:46.549 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:46.593 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:46.593 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:46.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:46.644 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:46.638+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:46.744 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:46.742+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:46.763 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 5, 0] out_osds: [7] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:37:46.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T00:37:46.792 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:46.792 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:46.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:46.845 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:46.842+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:46.947 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:46.942+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:46.971 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:46.971 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:46.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:47.047 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:47.042+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:47.148 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:47.146+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:47.153 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:47.154 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:47.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:47.249 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:47.246+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:47.326 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:47.326 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:47.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:47.385 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:47.382+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:47.450 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:47.446+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:47.463 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:47.463 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:47.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:47.551 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:47.546+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:47.599 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T00:37:47.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:37:47.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T00:37:47.652 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:47.650+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:47.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:47.752 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:47.746+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:47.798 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:47.798 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:47.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:47.857 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:47.854+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:47.954 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:47.950+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:47.972 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:47.972 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:47.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:48.055 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:48.050+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:48.147 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:48.147 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:48.156 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:48.154+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:48.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:48.258 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:48.254+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:48.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:48.359 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:48.354+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:48.424 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:48.424 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:48.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:48.459 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:48.454+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:48.557 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:48.557 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:48.560 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:48.554+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:48.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:48.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:48.602 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:37:48.617 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T00:37:48.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T00:37:48.693 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:48.690+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:48.732 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:48.732 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:48.736 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:48.737 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:48.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:48.754 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:48.754 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:48.761 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:48.758+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:48.771 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:48.771 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:48.788 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:48.788 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:48.809 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:48.809 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:48.836 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:48.836 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:48.857 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:48.857 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:48.861 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:48.858+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:48.872 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:48.872 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:48.872 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:48.872 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:48.872 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:48.877 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:48.877 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:48.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:48.985 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:48.982+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:49.063 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.819378172, 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:49.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:37:49.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:37:49.069 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.789610691000007, 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:49.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:37:49.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:37:49.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126791 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 112640~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.420045662000007, 2024-04-24T00:37:49.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074437884999999995, 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126791 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640034+0000", 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640044+0000", 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2660000000000007e-06 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640050+0000", 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713112+0000", 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073062031999999999 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713345+0000", 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000232616 2024-04-24T00:37:49.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714469+0000", 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011237669999999999 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126792 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 116736~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:49.078 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.420016888999996, 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075734135999999994, 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126792 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3399999999999999e-07 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640063+0000", 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7e-06 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640072+0000", 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0729999999999994e-06 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640084+0000", 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2486e-05 2024-04-24T00:37:49.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714588+0000", 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074503102000000002 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714763+0000", 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175885 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715794+0000", 2024-04-24T00:37:49.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001030256 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126793 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 120832~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.419983099000007, 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076915124000000001, 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.083 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126793 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640094+0000", 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640097+0000", 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9799999999999998e-06 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640107+0000", 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6229999999999992e-06 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640123+0000", 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5892999999999998e-05 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715884+0000", 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075761384000000001 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.716061+0000", 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176727 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717009+0000", 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094787400000000003 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126794 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 124928~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.419944419999993, 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078057396000000001, 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.087 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126794 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640133+0000", 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-07 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640136+0000", 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1e-06 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640145+0000", 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.516e-06 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640152+0000", 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717088+0000", 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076936302999999998 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717201+0000", 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112704 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718189+0000", 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098839299999999995 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126795 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 129024~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.419915533999998, 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079207986999999994, 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126795 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:49.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640162+0000", 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7400000000000003e-07 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640165+0000", 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1860000000000001e-06 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640174+0000", 2024-04-24T00:37:49.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5429999999999999e-06 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640181+0000", 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8029999999999999e-06 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718272+0000", 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078091277000000001 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718443+0000", 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171027 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.719369+0000", 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092547699999999996 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127300 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 229376~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.095 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.397659274999995, 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073789773000000003, 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127300 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662418+0000", 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.203e-06 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662439+0000", 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0713e-05 2024-04-24T00:37:49.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662460+0000", 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.105e-05 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662488+0000", 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7966e-05 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735368+0000", 2024-04-24T00:37:49.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072880167999999995 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735555+0000", 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018692399999999999 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736207+0000", 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065174899999999997 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127301 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 233472~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.397558316000001, 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074400210999999994, 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127301 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.100 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:49.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662520+0000", 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8700000000000001e-06 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662527+0000", 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7830000000000001e-06 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662549+0000", 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2379e-05 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662564+0000", 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.473e-05 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736281+0000", 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073717486999999998 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736386+0000", 2024-04-24T00:37:49.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104594 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736918+0000", 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053236799999999997 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127302 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.397490391000005, 2024-04-24T00:37:49.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075297446000000004, 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127302 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662587+0000", 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096e-06 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662608+0000", 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-04-24T00:37:49.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662624+0000", 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5563e-05 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662639+0000", 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4766999999999999e-05 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737013+0000", 2024-04-24T00:37:49.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074374061000000005 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737195+0000", 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182201 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737884+0000", 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068866800000000003 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127303 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.397416512999996, 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075989524000000003, 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127303 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.109 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662662+0000", 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5400000000000001e-06 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662668+0000", 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1970000000000001e-06 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662689+0000", 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0953000000000001e-05 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662703+0000", 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737962+0000", 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075258932000000001 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738074+0000", 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011173800000000001 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738649+0000", 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057545399999999998 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127304 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.397352179999999, 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076894431999999999, 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127304 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662725+0000", 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0359999999999999e-06 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662732+0000", 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6440000000000003e-06 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662751+0000", 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8583000000000001e-05 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662768+0000", 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7336000000000001e-05 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738744+0000", 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075976005999999999 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738924+0000", 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017996900000000001 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739619+0000", 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069485800000000002 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127305 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.117 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:49.118 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.397287904999999, 2024-04-24T00:37:49.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077646889999999996, 2024-04-24T00:37:49.118 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.118 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.118 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.118 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.118 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.118 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:49.058+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:49.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.119 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127305 2024-04-24T00:37:49.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.119 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662790+0000", 2024-04-24T00:37:49.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662796+0000", 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3659999999999997e-06 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662819+0000", 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2503000000000001e-05 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662834+0000", 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739755+0000", 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076920905999999997 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739876+0000", 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121445 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740435+0000", 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055911299999999997 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127306 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.397220341999997, 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078485103000000001, 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.123 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127306 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:49.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662857+0000", 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1599999999999999e-06 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662864+0000", 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3829999999999997e-06 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662883+0000", 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9476000000000001e-05 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662897+0000", 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3485999999999999e-05 2024-04-24T00:37:49.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740564+0000", 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077667768999999998 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740736+0000", 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017116700000000001 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741341+0000", 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060566199999999998 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127307 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.397160936999995, 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079342509000000006, 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127307 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662917+0000", 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662922+0000", 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662939+0000", 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833e-05 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662954+0000", 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741518+0000", 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078564118000000002 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741709+0000", 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191337 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.742258+0000", 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054904499999999996 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127559 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 243712~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.376376632000003, 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073184308000000003, 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127559 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:49.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683701+0000", 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2300000000000001e-06 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683724+0000", 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.321e-05 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683745+0000", 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.101e-05 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683761+0000", 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-04-24T00:37:49.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756124+0000", 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072363532999999994 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756327+0000", 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020239999999999999 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756884+0000", 2024-04-24T00:37:49.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055760300000000005 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127560 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 247808~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.376292892999999, 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073924043999999994, 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127560 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683785+0000", 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1829999999999999e-06 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683792+0000", 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2799999999999998e-06 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683810+0000", 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8076000000000001e-05 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683842+0000", 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756969+0000", 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073126925999999995 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757172+0000", 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020269900000000001 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757708+0000", 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053609999999999997 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127561 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 251904~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.376211565000006, 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074620636000000004, 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127561 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683866+0000", 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.127e-06 2024-04-24T00:37:49.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683873+0000", 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683893+0000", 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9300000000000002e-05 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683954+0000", 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1148999999999993e-05 2024-04-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757783+0000", 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073829025000000006 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757939+0000", 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582700000000001 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758486+0000", 2024-04-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054703200000000003 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127562 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 256000~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.376089969999995, 2024-04-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075112683, 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127562 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683988+0000", 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.443e-06 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684000+0000", 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.252e-05 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684025+0000", 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4956e-05 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684044+0000", 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8770000000000002e-05 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758566+0000", 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074522094999999997 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758669+0000", 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010247399999999999 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759099+0000", 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043042500000000001 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127563 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 260096~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.375996138000005, 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075791184999999997, 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127563 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684082+0000", 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684089+0000", 2024-04-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4889999999999998e-06 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684115+0000", 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5643000000000002e-05 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684133+0000", 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7842999999999998e-05 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759174+0000", 2024-04-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075041103999999997 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759329+0000", 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015504200000000001 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759872+0000", 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054272399999999996 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.155 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.155 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:49.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:49.163 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:49.158+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:49.266 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:49.262+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:49.290 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:49.291 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:49.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:49.393 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:49.390+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:49.468 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:49.466+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:49.478 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:49.478 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:49.478 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:49.478 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:49.478 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:49.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:49.581 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:49.578+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:49.594 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T00:37:49.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T00:37:49.669 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:49.666+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:49.692 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 216.47803382800001, 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:37:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:37:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:37:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:37:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 216.477682913, 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:37:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:37:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 216.47079459299999, 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:37:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 165.21251296700001, 2024-04-24T00:37:49.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:37:49.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:37:49.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 165.202236551, 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:49.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:49.719 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.74028574799999, 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:49.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:37:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.714251727, 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:37:49.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:37:49.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:37:49.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:37:49.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:37:49.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:37:49.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:37:49.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:37:49.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:37:49.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:37:49.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:37:49.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:37:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:37:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:37:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:37:49.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:49.733 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.50030881999999, 2024-04-24T00:37:49.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:37:49.734 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.734 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.734 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:49.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:49.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:37:49.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:37:49.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:37:49.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:37:49.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:37:49.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:37:49.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:37:49.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:37:49.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:37:49.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:37:49.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:37:49.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:37:49.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.743 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:37:49.743 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:49.743 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.490296876, 2024-04-24T00:37:49.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:37:49.744 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.744 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.744 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.744 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:49.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:49.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:37:49.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:37:49.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:37:49.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:37:49.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:37:49.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:37:49.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:37:49.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:37:49.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:37:49.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:37:49.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:37:49.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:37:49.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:37:49.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:37:49.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:37:49.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:37:49.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.754 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:37:49.754 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:49.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 126.216918014, 2024-04-24T00:37:49.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:37:49.755 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.755 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.755 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.756 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:37:49.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.756 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:49.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:49.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:37:49.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:37:49.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:37:49.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:37:49.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:37:49.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:37:49.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:37:49.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:37:49.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:37:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:37:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:37:49.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:37:49.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:49.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:37:49.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:49.764 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 126.20024956499999, 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:49.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.01435178200001, 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.769 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:37:49.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:37:49.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.330593425, 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.775 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:37:49.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.776 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:49.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:49.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:37:49.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:37:49.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:37:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:37:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:37:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:37:49.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:37:49.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:37:49.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:37:49.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:37:49.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:37:49.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:37:49.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.308376427, 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.784 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:37:49.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.003806589000007, 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.788 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:37:49.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:37:49.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.991181628999996, 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:37:49.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.641690142000002, 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:37:49.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:37:49.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.626671946999998, 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:49.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:37:49.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.733855544000001, 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:49.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:49.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:37:49.814 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.696503517, 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:49.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:37:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:37:49.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:49.818 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:49.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:49.820 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:49.814+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:49.921 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:49.918+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:49.954 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:49.954 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:49.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.954 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:173908 3.8 3.4c8948a8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-04-24T00:37:49.954 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.28552339300000001, 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.285547887, 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:49.955 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 173908 2024-04-24T00:37:49.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:49.956 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:49.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.956 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:49.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:49.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:49.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:49.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:49.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:49.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:49.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:49.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:49.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668860+0000", 2024-04-24T00:37:49.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7269999999999999e-06 2024-04-24T00:37:49.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:49.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668868+0000", 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3799999999999994e-06 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668889+0000", 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0619999999999999e-05 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:49.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:49.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668906+0000", 2024-04-24T00:37:49.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7303e-05 2024-04-24T00:37:49.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:49.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:49.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669040+0000", 2024-04-24T00:37:49.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134213 2024-04-24T00:37:49.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:49.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:49.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:49.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669057+0000", 2024-04-24T00:37:49.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7045999999999999e-05 2024-04-24T00:37:49.961 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:49.962 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:49.962 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:49.962 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:49.962 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:37:49.962 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:37:49.962 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:49.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:50.022 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:50.018+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:50.122 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:50.118+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:50.135 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:50.135 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:50.136 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:50.136 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:50.136 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:50.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:50.224 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:50.218+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:50.278 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:50.278 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:50.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:50.278 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 217.060484962, 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.279 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:37:50.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:37:50.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 202.63869926800001, 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.285 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:37:50.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.49657170200001, 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:37:50.290 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:37:50.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:37:50.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.35879992, 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:37:50.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:37:50.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:37:50.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:37:50.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.32131760199999, 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:37:50.302 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:37:50.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:37:50.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:37:50.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.65897173799999, 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.308 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.64911370199999, 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:37:50.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:50.319 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 132.62876409399999, 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:37:50.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:37:50.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:37:50.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 121.594718759, 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:37:50.325 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.329 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 121.59240892299999, 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:37:50.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 117.64469103099999, 2024-04-24T00:37:50.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:37:50.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:37:50.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:37:50.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.755301035, 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.342 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.711825238, 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.348 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:37:50.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:37:50.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.352 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 99.737851935999998, 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:37:50.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.759422479999998, 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:37:50.358 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:50.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:50.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:37:50.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:37:50.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:37:50.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:37:50.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:37:50.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:37:50.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:37:50.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:37:50.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:37:50.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:37:50.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:37:50.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:37:50.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:50.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:37:50.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:37:50.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:50.368 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.725420057999997, 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.372 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157093 3.13 3:cd0b7234:::rbd_data.115ee709dad1.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.388814822000001, 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12167272799999999, 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157093 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885996+0000", 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.426e-06 2024-04-24T00:37:50.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886003+0000", 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8859999999999998e-06 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886020+0000", 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6135000000000001e-05 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886027+0000", 2024-04-24T00:37:50.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3379999999999999e-06 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886099+0000", 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1802000000000001e-05 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886260+0000", 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016156800000000001 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.895327+0000", 2024-04-24T00:37:50.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090669270000000007 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007496+0000", 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112168567 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007520+0000", 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4646000000000001e-05 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007599+0000", 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8424999999999995e-05 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007668+0000", 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9008000000000001e-05 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:157093 3.13 e197/106 ondisk, result = 0)", 2024-04-24T00:37:50.378 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 43.357396784000002, 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090093925000000005, 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:50.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917416+0000", 2024-04-24T00:37:50.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:50.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917426+0000", 2024-04-24T00:37:50.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-04-24T00:37:50.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917465+0000", 2024-04-24T00:37:50.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9295999999999998e-05 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917484+0000", 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007420+0000", 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089936099000000005 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007435+0000", 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4966999999999999e-05 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007507+0000", 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2206000000000006e-05 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.384 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169765 3.13 3:c9e19d2e:::rbd_data.115ee24d4bb.0000000000000011:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.415031184, 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096474246999999999, 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169765 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859781+0000", 2024-04-24T00:37:50.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859792+0000", 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859819+0000", 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7025999999999998e-05 2024-04-24T00:37:50.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859833+0000", 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3763e-05 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859956+0000", 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122989 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.860147+0000", 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190843 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.866555+0000", 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064084540000000001 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.955923+0000", 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089367567999999994 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956056+0000", 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013292600000000001 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956157+0000", 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101029 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956253+0000", 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5921999999999994e-05 2024-04-24T00:37:50.390 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169765 3.13 e210/106 ondisk, result = 0)", 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.39574975, 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077208636999999997, 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879060+0000", 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879062+0000", 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879072+0000", 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0149999999999999e-05 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879106+0000", 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3841999999999999e-05 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.879126+0000", 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9264000000000001e-05 2024-04-24T00:37:50.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956025+0000", 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076899281999999999 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956045+0000", 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9902999999999999e-05 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956269+0000", 2024-04-24T00:37:50.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000224012 2024-04-24T00:37:50.395 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.395 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.395 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.395 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.395 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.395 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:50.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:50.397 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:50.390+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:50.497 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:50.494+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:50.568 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:50.568 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:50.569 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:50.569 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:50.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:50.594 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T00:37:50.599 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:50.594+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:50.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:37:50.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-24T00:37:50.713 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:50.714 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:50.714 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:50.714 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:50.714 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:50.728 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:50.729 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:50.726+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:50.800 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:50.794+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 203.14050096099999, 2024-04-24T00:37:50.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:37:50.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 203.14040303600001, 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.858 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:37:50.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:37:50.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:37:50.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:37:50.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:37:50.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.865 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.866 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:50.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:50.866 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.78102350399999, 2024-04-24T00:37:50.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:37:50.866 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.866 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.866 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.867 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.867 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:37:50.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.867 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:50.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:50.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:37:50.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:37:50.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:37:50.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:37:50.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:37:50.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:37:50.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:37:50.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:37:50.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:37:50.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:37:50.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:37:50.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:37:50.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:50.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:37:50.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:37:50.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:37:50.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:37:50.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:37:50.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:37:50.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 166.37027937799999, 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:37:50.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:37:50.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 166.34501077199999, 2024-04-24T00:37:50.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:37:50.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.886 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 152.49180705200001, 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:37:50.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:37:50.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:37:50.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 152.45920193800001, 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 152.05462724399999, 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:37:50.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:37:50.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:50.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.902 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.99678701400001, 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:37:50.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:37:50.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 105.17647711399999, 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:37:50.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:37:50.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:37:50.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:37:50.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.915 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 105.146835061, 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:37:50.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 105.00031964999999, 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.920 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.211547092, 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.926 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:50.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:37:50.931 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 100.188403679, 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:37:50.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:37:50.935 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.315094931000004, 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.936 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.937 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:37:50.938 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:37:50.939 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.940 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.941 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 90.262412654000002, 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:50.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:37:50.945 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:166979 3.5 3:a3485662:::rbd_data.115e4ca9eff1.0000000000000010:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.862680701000002, 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.145524811, 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.946 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 166979 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986190+0000", 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.323e-06 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986212+0000", 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1183999999999999e-05 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986246+0000", 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4409999999999998e-05 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986265+0000", 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986437+0000", 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017130199999999999 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986581+0000", 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014409500000000001 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.997390+0000", 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010809456 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131432+0000", 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.134041997 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131521+0000", 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9024999999999996e-05 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131625+0000", 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103923 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131713+0000", 2024-04-24T00:37:50.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7672999999999999e-05 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:166979 3.5 e206/56 ondisk, result = 0)", 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.819145732999999, 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10200487, 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029726+0000", 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4899999999999999e-06 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.953 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029738+0000", 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2483000000000001e-05 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029784+0000", 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5920000000000001e-05 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029809+0000", 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5193000000000001e-05 2024-04-24T00:37:50.954 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131491+0000", 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101681585 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131510+0000", 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9256999999999999e-05 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.955 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131728+0000", 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021794199999999999 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169782 3.d 3:b3c6180a:::rbd_data.115ee24d4bb.000000000000000f:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.716023193000002, 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.117826449, 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:50.956 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169782 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:50.957 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132848+0000", 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0770000000000002e-06 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132859+0000", 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.958 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132887+0000", 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8070000000000001e-05 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132908+0000", 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.09e-05 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133079+0000", 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017107900000000001 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.959 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133247+0000", 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168138 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.141250+0000", 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080027740000000007 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250223+0000", 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108972528 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.960 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250437+0000", 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021457900000000001 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250540+0000", 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102682 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250672+0000", 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013205899999999999 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.961 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169782 3.d e210/106 ondisk, result = 0)", 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 28.687106861, 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088930206999999997, 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:50.962 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161764+0000", 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3259999999999998e-06 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161773+0000", 2024-04-24T00:37:50.963 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5329999999999994e-06 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161803+0000", 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9867000000000001e-05 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161825+0000", 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2082999999999998e-05 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250323+0000", 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088498544999999998 2024-04-24T00:37:50.964 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250342+0000", 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8983000000000001e-05 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250692+0000", 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034987000000000001 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:50.965 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:50.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:50.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:50.968 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:50.962+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:51.068 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:51.066+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:51.130 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:51.131 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:51.134 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:51.134 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:51.134 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:51.134 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:51.149 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:51.149 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:51.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:51.168 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:51.169 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:51.169 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:51.162+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:51.186 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:51.187 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:51.206 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:51.206 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:51.226 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:51.226 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:51.245 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:51.246 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:51.267 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:51.267 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:51.269 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:51.266+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:51.327 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:51.327 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:51.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:51.371 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:51.366+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:51.471 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:51.466+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:51.515 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:51.515 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:51.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:51.576 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:51.574+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:51.597 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-04-24T00:37:51.603 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:37:51.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T00:37:51.659 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:51.659 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:51.672 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:51.670+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:51.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:51.774 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:51.770+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:51.852 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:51.853 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:51.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:51.874 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:51.870+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:51.975 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:51.970+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:51.995 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:51.995 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:52.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:52.076 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:52.070+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:52.145 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:52.145 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:52.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:52.205 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:52.202+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:52.278 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:52.274+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:52.332 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:52.332 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:52.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:52.379 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:52.378+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:52.465 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:52.465 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:52.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:52.480 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:52.474+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:52.581 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:52.574+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:52.600 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T00:37:52.619 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:52.620 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:52.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T00:37:52.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:52.686 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:52.682+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:52.782 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:52.778+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:52.806 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:52.806 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:52.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:52.883 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:52.878+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:52.983 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:52.978+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:52.985 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:52.985 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:52.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:53.085 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:53.082+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:53.163 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:53.163 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:53.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:53.221 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:53.218+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:53.285 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:53.282+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:53.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:53.350 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:53.351 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:53.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:53.417 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:53.414+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:53.433 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:53.433 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:53.450 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:53.451 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:53.468 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:53.468 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:53.485 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:53.488 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:53.482+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:106775 3.0 3:06726de3:::rbd_data.115ec2448e7a.0000000000000006:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.254182580000005, 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123187456, 2024-04-24T00:37:53.498 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106775 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240698+0000", 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240700+0000", 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240710+0000", 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4269999999999997e-06 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240737+0000", 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7019e-05 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240755+0000", 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8836000000000001e-05 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240841+0000", 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5217999999999996e-05 2024-04-24T00:37:53.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.240931+0000", 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0358999999999993e-05 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.247487+0000", 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0065560089999999998 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363695+0000", 2024-04-24T00:37:53.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.116207754 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363770+0000", 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5282000000000002e-05 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363833+0000", 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2557000000000004e-05 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363886+0000", 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3208999999999998e-05 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:106775 3.0 e154/123 ondisk, result = 0)", 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.224415098999998, 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428945999999999, 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.504 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270466+0000", 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270468+0000", 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-04-24T00:37:53.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270479+0000", 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270520+0000", 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1075000000000002e-05 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.270543+0000", 2024-04-24T00:37:53.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3349999999999998e-05 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363753+0000", 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093209502999999999 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363764+0000", 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1027e-05 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:20.363895+0000", 2024-04-24T00:37:53.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131104 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126791 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 112640~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.854850069999998, 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074437884999999995, 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126791 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640031+0000", 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1399999999999997e-07 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640034+0000", 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640044+0000", 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2660000000000007e-06 2024-04-24T00:37:53.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640050+0000", 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713112+0000", 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073062031999999999 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.713345+0000", 2024-04-24T00:37:53.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000232616 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714469+0000", 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011237669999999999 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126792 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 116736~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:53.512 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.854821297000001, 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075734135999999994, 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126792 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640060+0000", 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3399999999999999e-07 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640063+0000", 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7e-06 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640072+0000", 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0729999999999994e-06 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640084+0000", 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2486e-05 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714588+0000", 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074503102000000002 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.714763+0000", 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175885 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715794+0000", 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001030256 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126793 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 120832~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.854787506999998, 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076915124000000001, 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.517 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126793 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640093+0000", 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640094+0000", 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640097+0000", 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9799999999999998e-06 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640107+0000", 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6229999999999992e-06 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640123+0000", 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5892999999999998e-05 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.715884+0000", 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075761384000000001 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.716061+0000", 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176727 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717009+0000", 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094787400000000003 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126794 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 124928~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.854748827999998, 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078057396000000001, 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126794 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640132+0000", 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640133+0000", 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-07 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640136+0000", 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1e-06 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640145+0000", 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.516e-06 2024-04-24T00:37:53.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640152+0000", 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.72e-06 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717088+0000", 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076936302999999998 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.717201+0000", 2024-04-24T00:37:53.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112704 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718189+0000", 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098839299999999995 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:126795 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 129024~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.854719942000003, 2024-04-24T00:37:53.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079207986999999994, 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 126795 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640161+0000", 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640162+0000", 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7400000000000003e-07 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640165+0000", 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1860000000000001e-06 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640174+0000", 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5429999999999999e-06 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.640181+0000", 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8029999999999999e-06 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718272+0000", 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078091277000000001 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.718443+0000", 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171027 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.719369+0000", 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092547699999999996 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127300 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 229376~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.832463683, 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073789773000000003, 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.530 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127300 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662417+0000", 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662418+0000", 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.203e-06 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662439+0000", 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0713e-05 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662460+0000", 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.105e-05 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662488+0000", 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7966e-05 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735368+0000", 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072880167999999995 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.735555+0000", 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018692399999999999 2024-04-24T00:37:53.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736207+0000", 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065174899999999997 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127301 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 233472~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.832362724000006, 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074400210999999994, 2024-04-24T00:37:53.534 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127301 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662518+0000", 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662520+0000", 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8700000000000001e-06 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662527+0000", 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7830000000000001e-06 2024-04-24T00:37:53.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662549+0000", 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2379e-05 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662564+0000", 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.473e-05 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736281+0000", 2024-04-24T00:37:53.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073717486999999998 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736386+0000", 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104594 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.736918+0000", 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053236799999999997 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127302 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 237568~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.832294798999996, 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075297446000000004, 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127302 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662586+0000", 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662587+0000", 2024-04-24T00:37:53.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.096e-06 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662608+0000", 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662624+0000", 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5563e-05 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662639+0000", 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4766999999999999e-05 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737013+0000", 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074374061000000005 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737195+0000", 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182201 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737884+0000", 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068866800000000003 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127303 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 241664~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.832220921000001, 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075989524000000003, 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127303 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662660+0000", 2024-04-24T00:37:53.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662662+0000", 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5400000000000001e-06 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662668+0000", 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1970000000000001e-06 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662689+0000", 2024-04-24T00:37:53.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0953000000000001e-05 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662703+0000", 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.737962+0000", 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075258932000000001 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738074+0000", 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011173800000000001 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738649+0000", 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057545399999999998 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127304 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 245760~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:53.547 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.832156588000004, 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076894431999999999, 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127304 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:53.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662724+0000", 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662725+0000", 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0359999999999999e-06 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662732+0000", 2024-04-24T00:37:53.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6440000000000003e-06 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662751+0000", 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8583000000000001e-05 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662768+0000", 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7336000000000001e-05 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738744+0000", 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075976005999999999 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.738924+0000", 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017996900000000001 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739619+0000", 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069485800000000002 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127305 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 249856~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.832092313000004, 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077646889999999996, 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.552 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127305 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662789+0000", 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662790+0000", 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0839999999999999e-06 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662796+0000", 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3659999999999997e-06 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662819+0000", 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2503000000000001e-05 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662834+0000", 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739755+0000", 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076920905999999997 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.739876+0000", 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121445 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740435+0000", 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055911299999999997 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127306 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 253952~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.832024750000002, 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078485103000000001, 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127306 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662856+0000", 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662857+0000", 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1599999999999999e-06 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662864+0000", 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3829999999999997e-06 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662883+0000", 2024-04-24T00:37:53.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9476000000000001e-05 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662897+0000", 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3485999999999999e-05 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740564+0000", 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.077667768999999998 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.740736+0000", 2024-04-24T00:37:53.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017116700000000001 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741341+0000", 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060566199999999998 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127307 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 258048~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.831965345, 2024-04-24T00:37:53.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.079342509000000006, 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127307 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662916+0000", 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662917+0000", 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9300000000000006e-07 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662922+0000", 2024-04-24T00:37:53.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662939+0000", 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6833e-05 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.662954+0000", 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741518+0000", 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078564118000000002 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.741709+0000", 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191337 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.742258+0000", 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054904499999999996 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127559 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 243712~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.811181039999994, 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073184308000000003, 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127559 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683700+0000", 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683701+0000", 2024-04-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2300000000000001e-06 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683724+0000", 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.321e-05 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683745+0000", 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.101e-05 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683761+0000", 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756124+0000", 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072363532999999994 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756327+0000", 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020239999999999999 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756884+0000", 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055760300000000005 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127560 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 247808~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.811097301000004, 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073924043999999994, 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.572 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127560 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683784+0000", 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683785+0000", 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1829999999999999e-06 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683792+0000", 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2799999999999998e-06 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683810+0000", 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8076000000000001e-05 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683842+0000", 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.756969+0000", 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073126925999999995 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757172+0000", 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020269900000000001 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757708+0000", 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053609999999999997 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127561 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 251904~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.811015972999996, 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074620636000000004, 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127561 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683865+0000", 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683866+0000", 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.127e-06 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683873+0000", 2024-04-24T00:37:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1760000000000004e-06 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683893+0000", 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9300000000000002e-05 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683954+0000", 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1148999999999993e-05 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757783+0000", 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073829025000000006 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.757939+0000", 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015582700000000001 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758486+0000", 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054703200000000003 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127562 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 256000~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.810894378, 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075112683, 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127562 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.581 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683987+0000", 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.683988+0000", 2024-04-24T00:37:53.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.443e-06 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684000+0000", 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.252e-05 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684025+0000", 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4956e-05 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684044+0000", 2024-04-24T00:37:53.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8770000000000002e-05 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758566+0000", 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074522094999999997 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.758669+0000", 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010247399999999999 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759099+0000", 2024-04-24T00:37:53.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043042500000000001 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:127563 3.0 3:0238e5bb:::rbd_data.115ee44a9909.0000000000000005:46 [read 260096~2048 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e178)", 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.810800545999996, 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075791184999999997, 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:53.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127563 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684080+0000", 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:53.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684082+0000", 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3400000000000001e-06 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684089+0000", 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4889999999999998e-06 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684115+0000", 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5643000000000002e-05 2024-04-24T00:37:53.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.684133+0000", 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7842999999999998e-05 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759174+0000", 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075041103999999997 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759329+0000", 2024-04-24T00:37:53.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015504200000000001 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:44.759872+0000", 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054272399999999996 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:53.589 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:53.590 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:53.591 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:53.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:53.592 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:53.592 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:53.592 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:53.593 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:53.593 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:53.594 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:53.594 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:53.586+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:53.601 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T00:37:53.631 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T00:37:53.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:37:53.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:37:53.691 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:53.686+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:53.726 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:53.726 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:53.726 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:53.726 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:53.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:53.792 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:53.786+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:53.892 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:53.890+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:53.905 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:53.905 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:53.905 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:53.905 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:53.905 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:53.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:53.975 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:37:53.976 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:37:53.992 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-04-24T00:37:53.713602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":903120317,"num_objects":539,"num_object_clones":0,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":181209,"num_read_kb":1541747,"num_write":30131,"num_write_kb":1020305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4494,"num_bytes_recovered":320728151,"num_keys_recovered":439,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21549,"ondisk_log_size":21549,"up":82,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":91,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":445236,"kb_used_data":47180,"kb_used_omap":15,"kb_used_meta":397936,"kb_avail":660157644,"statfs":{"total":676457349120,"available":676001427456,"internally_reserved":0,"allocated":48312320,"data_stored":230131497,"data_compressed":8709052,"data_compressed_allocated":22179840,"data_compressed_original":217054327,"omap_allocated":15996,"internal_metadata":407486852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001366"},"pg_stats":[{"pgid":"3.1f","version":"231'831","reported_seq":5674,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594994+0000","last_change":"2024-04-24T00:37:47.623538+0000","last_active":"2024-04-24T00:37:49.594994+0000","last_peered":"2024-04-24T00:37:49.594994+0000","last_clean":"2024-04-24T00:37:49.594994+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:37:49.594994+0000","last_undegraded":"2024-04-24T00:37:49.594994+0000","last_fullsized":"2024-04-24T00:37:49.594994+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:01:23.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019480669999999999,"stat_sum":{"num_bytes":52367280,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6142,"num_read_kb":55851,"num_write":1308,"num_write_kb":38951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1e","version":"231'977","reported_seq":5291,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.620237+0000","last_change":"2024-04-24T00:37:46.609936+0000","last_active":"2024-04-24T00:37:48.620237+0000","last_peered":"2024-04-24T00:37:48.620237+0000","last_clean":"2024-04-24T00:37:48.620237+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:37:48.620237+0000","last_undegraded":"2024-04-24T00:37:48.620237+0000","last_fullsized":"2024-04-24T00:37:48.620237+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:42.516498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003187456,"stat_sum":{"num_bytes":21004221,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7576,"num_read_kb":34899,"num_write":1269,"num_write_kb":26860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1d","version":"232'552","reported_seq":6735,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612940+0000","last_change":"2024-04-24T00:37:46.614489+0000","last_active":"2024-04-24T00:37:48.612940+0000","last_peered":"2024-04-24T00:37:48.612940+0000","last_clean":"2024-04-24T00:37:48.612940+0000","last_became_active":"2024-04-24T00:34:28.726196+0000","last_became_peered":"2024-04-24T00:34:28.726196+0000","last_unstale":"2024-04-24T00:37:48.612940+0000","last_undegraded":"2024-04-24T00:37:48.612940+0000","last_fullsized":"2024-04-24T00:37:48.612940+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:00:13.573517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001963238,"stat_sum":{"num_bytes":32555141,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7132,"num_read_kb":66702,"num_write":793,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":458933,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1c","version":"235'712","reported_seq":5393,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.611973+0000","last_change":"2024-04-24T00:37:46.613357+0000","last_active":"2024-04-24T00:37:48.611973+0000","last_peered":"2024-04-24T00:37:48.611973+0000","last_clean":"2024-04-24T00:37:48.611973+0000","last_became_active":"2024-04-24T00:34:31.747052+0000","last_became_peered":"2024-04-24T00:34:31.747052+0000","last_unstale":"2024-04-24T00:37:48.611973+0000","last_undegraded":"2024-04-24T00:37:48.611973+0000","last_fullsized":"2024-04-24T00:37:48.611973+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:11.238051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001537114,"stat_sum":{"num_bytes":32968971,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4941,"num_read_kb":61078,"num_write":909,"num_write_kb":35121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":949407,"num_keys_recovered":27,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1b","version":"231'455","reported_seq":4234,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612191+0000","last_change":"2024-04-24T00:37:46.611866+0000","last_active":"2024-04-24T00:37:48.612191+0000","last_peered":"2024-04-24T00:37:48.612191+0000","last_clean":"2024-04-24T00:37:48.612191+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:37:48.612191+0000","last_undegraded":"2024-04-24T00:37:48.612191+0000","last_fullsized":"2024-04-24T00:37:48.612191+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:08.897008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056302700000000002,"stat_sum":{"num_bytes":26704680,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3990,"num_read_kb":45455,"num_write":567,"num_write_kb":27383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1a","version":"232'374","reported_seq":5487,"reported_epoch":235,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:47.611481+0000","last_change":"2024-04-24T00:37:48.608289+0000","last_active":"2024-04-24T00:37:47.611481+0000","last_peered":"2024-04-24T00:37:46.679778+0000","last_clean":"2024-04-24T00:37:46.679778+0000","last_became_active":"2024-04-24T00:35:47.582331+0000","last_became_peered":"2024-04-24T00:35:47.582331+0000","last_unstale":"2024-04-24T00:37:47.611481+0000","last_undegraded":"2024-04-24T00:37:47.611481+0000","last_fullsized":"2024-04-24T00:37:47.611481+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048149600000000001,"stat_sum":{"num_bytes":9349759,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4387,"num_read_kb":37212,"num_write":485,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":1449843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.19","version":"228'680","reported_seq":5653,"reported_epoch":234,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:46.604822+0000","last_change":"2024-04-24T00:37:48.613702+0000","last_active":"2024-04-24T00:37:48.613702+0000","last_peered":"2024-04-24T00:37:46.604822+0000","last_clean":"2024-04-24T00:37:46.604822+0000","last_became_active":"2024-04-24T00:35:47.589674+0000","last_became_peered":"2024-04-24T00:35:47.589674+0000","last_unstale":"2024-04-24T00:37:46.604822+0000","last_undegraded":"2024-04-24T00:37:46.604822+0000","last_fullsized":"2024-04-24T00:37:46.604822+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000930561,"stat_sum":{"num_bytes":24604173,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5924,"num_read_kb":40380,"num_write":975,"num_write_kb":29815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":868041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":1,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.18","version":"235'537","reported_seq":4709,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594755+0000","last_change":"2024-04-24T00:37:46.611758+0000","last_active":"2024-04-24T00:37:49.594755+0000","last_peered":"2024-04-24T00:37:49.594755+0000","last_clean":"2024-04-24T00:37:49.594755+0000","last_became_active":"2024-04-24T00:37:46.608136+0000","last_became_peered":"2024-04-24T00:37:46.608136+0000","last_unstale":"2024-04-24T00:37:49.594755+0000","last_undegraded":"2024-04-24T00:37:49.594755+0000","last_fullsized":"2024-04-24T00:37:49.594755+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:15.418243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014921719999999999,"stat_sum":{"num_bytes":28085793,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":584,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.17","version":"231'322","reported_seq":4593,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.596110+0000","last_change":"2024-04-24T00:37:46.618751+0000","last_active":"2024-04-24T00:37:49.596110+0000","last_peered":"2024-04-24T00:37:49.596110+0000","last_clean":"2024-04-24T00:37:49.596110+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:37:49.596110+0000","last_undegraded":"2024-04-24T00:37:49.596110+0000","last_fullsized":"2024-04-24T00:37:49.596110+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:16.564344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014710190000000001,"stat_sum":{"num_bytes":14945483,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":417,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.16","version":"231'670","reported_seq":6319,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612865+0000","last_change":"2024-04-24T00:37:46.605862+0000","last_active":"2024-04-24T00:37:48.612865+0000","last_peered":"2024-04-24T00:37:48.612865+0000","last_clean":"2024-04-24T00:37:48.612865+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:37:48.612865+0000","last_undegraded":"2024-04-24T00:37:48.612865+0000","last_fullsized":"2024-04-24T00:37:48.612865+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:28:04.964561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064798799999999995,"stat_sum":{"num_bytes":32925991,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7147,"num_read_kb":50536,"num_write":1023,"num_write_kb":38907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.15","version":"231'613","reported_seq":6812,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612606+0000","last_change":"2024-04-24T00:37:46.604886+0000","last_active":"2024-04-24T00:37:48.612606+0000","last_peered":"2024-04-24T00:37:48.612606+0000","last_clean":"2024-04-24T00:37:48.612606+0000","last_became_active":"2024-04-24T00:35:28.725126+0000","last_became_peered":"2024-04-24T00:35:28.725126+0000","last_unstale":"2024-04-24T00:37:48.612606+0000","last_undegraded":"2024-04-24T00:37:48.612606+0000","last_fullsized":"2024-04-24T00:37:48.612606+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:18:28.502951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056773200000000005,"stat_sum":{"num_bytes":36441735,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":878,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":2312722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.14","version":"228'476","reported_seq":4986,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607048+0000","last_change":"2024-04-24T00:37:46.610711+0000","last_active":"2024-04-24T00:37:48.607048+0000","last_peered":"2024-04-24T00:37:48.607048+0000","last_clean":"2024-04-24T00:37:48.607048+0000","last_became_active":"2024-04-24T00:35:28.722992+0000","last_became_peered":"2024-04-24T00:35:28.722992+0000","last_unstale":"2024-04-24T00:37:48.607048+0000","last_undegraded":"2024-04-24T00:37:48.607048+0000","last_fullsized":"2024-04-24T00:37:48.607048+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:45:17.421114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058891000000000004,"stat_sum":{"num_bytes":23091924,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4140,"num_read_kb":41580,"num_write":565,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1929866,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":449,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.619922+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:37:48.619922+0000","last_peered":"2024-04-24T00:37:48.619922+0000","last_clean":"2024-04-24T00:37:48.619922+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:37:48.619922+0000","last_undegraded":"2024-04-24T00:37:48.619922+0000","last_fullsized":"2024-04-24T00:37:48.619922+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"232'529","reported_seq":6038,"reported_epoch":237,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-24T00:37:49.599990+0000","last_change":"2024-04-24T00:37:49.631656+0000","last_active":"2024-04-24T00:37:47.612198+0000","last_peered":"2024-04-24T00:37:46.617891+0000","last_clean":"2024-04-24T00:37:46.617891+0000","last_became_active":"2024-04-24T00:37:46.611693+0000","last_became_peered":"2024-04-24T00:37:46.611693+0000","last_unstale":"2024-04-24T00:37:49.599990+0000","last_undegraded":"2024-04-24T00:37:49.599990+0000","last_fullsized":"2024-04-24T00:37:49.595246+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001201978,"stat_sum":{"num_bytes":48215459,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5114,"num_read_kb":58057,"num_write":872,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":44596023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":24}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.624832+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:37:48.624832+0000","last_peered":"2024-04-24T00:37:48.624832+0000","last_clean":"2024-04-24T00:37:48.624832+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:37:48.624832+0000","last_undegraded":"2024-04-24T00:37:48.624832+0000","last_fullsized":"2024-04-24T00:37:48.624832+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"231'580","reported_seq":4521,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.595609+0000","last_change":"2024-04-24T00:37:47.621693+0000","last_active":"2024-04-24T00:37:49.595609+0000","last_peered":"2024-04-24T00:37:49.595609+0000","last_clean":"2024-04-24T00:37:49.595609+0000","last_became_active":"2024-04-24T00:37:47.617077+0000","last_became_peered":"2024-04-24T00:37:47.617077+0000","last_unstale":"2024-04-24T00:37:49.595609+0000","last_undegraded":"2024-04-24T00:37:49.595609+0000","last_fullsized":"2024-04-24T00:37:49.595609+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:08:32.489010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002332027,"stat_sum":{"num_bytes":43812937,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4338,"num_read_kb":41656,"num_write":809,"num_write_kb":26241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.595036+0000","last_change":"2024-04-24T00:37:42.650295+0000","last_active":"2024-04-24T00:37:49.595036+0000","last_peered":"2024-04-24T00:37:49.595036+0000","last_clean":"2024-04-24T00:37:49.595036+0000","last_became_active":"2024-04-24T00:37:42.649872+0000","last_became_peered":"2024-04-24T00:37:42.649872+0000","last_unstale":"2024-04-24T00:37:49.595036+0000","last_undegraded":"2024-04-24T00:37:49.595036+0000","last_fullsized":"2024-04-24T00:37:49.595036+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:39:37.552779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"234'495","reported_seq":3892,"reported_epoch":236,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:48.613409+0000","last_change":"2024-04-24T00:37:48.613409+0000","last_active":"2024-04-24T00:37:48.613409+0000","last_peered":"2024-04-24T00:37:46.697143+0000","last_clean":"2024-04-24T00:37:46.697143+0000","last_became_active":"2024-04-24T00:35:47.586287+0000","last_became_peered":"2024-04-24T00:35:47.586287+0000","last_unstale":"2024-04-24T00:37:48.613409+0000","last_undegraded":"2024-04-24T00:37:48.613409+0000","last_fullsized":"2024-04-24T00:37:48.613409+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082295399999999998,"stat_sum":{"num_bytes":23735278,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3403,"num_read_kb":34380,"num_write":703,"num_write_kb":26463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1579365,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.624738+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:37:48.624738+0000","last_peered":"2024-04-24T00:37:48.624738+0000","last_clean":"2024-04-24T00:37:48.624738+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:37:48.624738+0000","last_undegraded":"2024-04-24T00:37:48.624738+0000","last_fullsized":"2024-04-24T00:37:48.624738+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"231'616","reported_seq":5955,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612474+0000","last_change":"2024-04-24T00:37:46.615072+0000","last_active":"2024-04-24T00:37:48.612474+0000","last_peered":"2024-04-24T00:37:48.612474+0000","last_clean":"2024-04-24T00:37:48.612474+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:37:48.612474+0000","last_undegraded":"2024-04-24T00:37:48.612474+0000","last_fullsized":"2024-04-24T00:37:48.612474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:45:18.835496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002440181,"stat_sum":{"num_bytes":34823236,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5966,"num_read_kb":55481,"num_write":794,"num_write_kb":40841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.3","version":"235'480","reported_seq":6213,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607853+0000","last_change":"2024-04-24T00:37:47.629526+0000","last_active":"2024-04-24T00:37:48.607853+0000","last_peered":"2024-04-24T00:37:48.607853+0000","last_clean":"2024-04-24T00:37:48.607853+0000","last_became_active":"2024-04-24T00:37:47.623000+0000","last_became_peered":"2024-04-24T00:37:47.623000+0000","last_unstale":"2024-04-24T00:37:48.607853+0000","last_undegraded":"2024-04-24T00:37:48.607853+0000","last_fullsized":"2024-04-24T00:37:48.607853+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:41:21.537719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026935660000000001,"stat_sum":{"num_bytes":3134459,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4940,"num_read_kb":51361,"num_write":642,"num_write_kb":30821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":29414668,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":375,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.611822+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:37:48.611822+0000","last_peered":"2024-04-24T00:37:48.611822+0000","last_clean":"2024-04-24T00:37:48.611822+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:37:48.611822+0000","last_undegraded":"2024-04-24T00:37:48.611822+0000","last_fullsized":"2024-04-24T00:37:48.611822+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"231'595","reported_seq":7557,"reported_epoch":234,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:46.603085+0000","last_change":"2024-04-24T00:37:48.614146+0000","last_active":"2024-04-24T00:37:48.614146+0000","last_peered":"2024-04-24T00:37:46.603085+0000","last_clean":"2024-04-24T00:37:46.603085+0000","last_became_active":"2024-04-24T00:35:47.586354+0000","last_became_peered":"2024-04-24T00:35:47.586354+0000","last_unstale":"2024-04-24T00:37:46.603085+0000","last_undegraded":"2024-04-24T00:37:46.603085+0000","last_fullsized":"2024-04-24T00:37:46.603085+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064418099999999996,"stat_sum":{"num_bytes":15384600,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7270,"num_read_kb":65726,"num_write":672,"num_write_kb":34854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3429938,"num_keys_recovered":22,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":1,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":444,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.614774+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:37:48.614774+0000","last_peered":"2024-04-24T00:37:48.614774+0000","last_clean":"2024-04-24T00:37:48.614774+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:37:48.614774+0000","last_undegraded":"2024-04-24T00:37:48.614774+0000","last_fullsized":"2024-04-24T00:37:48.614774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"231'426","reported_seq":4103,"reported_epoch":236,"state":"peering","last_fresh":"2024-04-24T00:37:48.610467+0000","last_change":"2024-04-24T00:37:49.596425+0000","last_active":"2024-04-24T00:37:49.596425+0000","last_peered":"2024-04-24T00:37:48.610467+0000","last_clean":"2024-04-24T00:37:48.610467+0000","last_became_active":"2024-04-24T00:37:45.607305+0000","last_became_peered":"2024-04-24T00:37:45.607305+0000","last_unstale":"2024-04-24T00:37:48.610467+0000","last_undegraded":"2024-04-24T00:37:48.610467+0000","last_fullsized":"2024-04-24T00:37:48.610467+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033054920000000002,"stat_sum":{"num_bytes":26338783,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2119,"num_read_kb":33775,"num_write":541,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":27566656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.622167+0000","last_change":"2024-04-24T00:37:48.620512+0000","last_active":"2024-04-24T00:37:48.622167+0000","last_peered":"2024-04-24T00:37:48.622167+0000","last_clean":"2024-04-24T00:37:48.622167+0000","last_became_active":"2024-04-24T00:37:48.620159+0000","last_became_peered":"2024-04-24T00:37:48.620159+0000","last_unstale":"2024-04-24T00:37:48.622167+0000","last_undegraded":"2024-04-24T00:37:48.622167+0000","last_fullsized":"2024-04-24T00:37:48.622167+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:47.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"231'557","reported_seq":2562,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612119+0000","last_change":"2024-04-24T00:37:46.613641+0000","last_active":"2024-04-24T00:37:48.612119+0000","last_peered":"2024-04-24T00:37:48.612119+0000","last_clean":"2024-04-24T00:37:48.612119+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:37:48.612119+0000","last_undegraded":"2024-04-24T00:37:48.612119+0000","last_fullsized":"2024-04-24T00:37:48.612119+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:10.421466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019991789999999998,"stat_sum":{"num_bytes":26453257,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2553,"num_read_kb":24145,"num_write":731,"num_write_kb":27938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":375,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.611902+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:37:48.611902+0000","last_peered":"2024-04-24T00:37:48.611902+0000","last_clean":"2024-04-24T00:37:48.611902+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:37:48.611902+0000","last_undegraded":"2024-04-24T00:37:48.611902+0000","last_fullsized":"2024-04-24T00:37:48.611902+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":405,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594456+0000","last_change":"2024-04-24T00:37:45.610754+0000","last_active":"2024-04-24T00:37:49.594456+0000","last_peered":"2024-04-24T00:37:49.594456+0000","last_clean":"2024-04-24T00:37:49.594456+0000","last_became_active":"2024-04-24T00:37:45.609794+0000","last_became_peered":"2024-04-24T00:37:45.609794+0000","last_unstale":"2024-04-24T00:37:49.594456+0000","last_undegraded":"2024-04-24T00:37:49.594456+0000","last_fullsized":"2024-04-24T00:37:49.594456+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:27.857798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"231'861","reported_seq":6617,"reported_epoch":234,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:46.607508+0000","last_change":"2024-04-24T00:37:48.607922+0000","last_active":"2024-04-24T00:37:48.607922+0000","last_peered":"2024-04-24T00:37:46.607508+0000","last_clean":"2024-04-24T00:37:46.607508+0000","last_became_active":"2024-04-24T00:35:54.570587+0000","last_became_peered":"2024-04-24T00:35:54.570587+0000","last_unstale":"2024-04-24T00:37:46.607508+0000","last_undegraded":"2024-04-24T00:37:46.607508+0000","last_fullsized":"2024-04-24T00:37:46.607508+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":861,"log_dups_size":0,"ondisk_log_size":861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031730159999999999,"stat_sum":{"num_bytes":31041185,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7207,"num_read_kb":68772,"num_write":1181,"num_write_kb":32528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":5566512,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":1,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.9","version":"232'932","reported_seq":4792,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607046+0000","last_change":"2024-04-24T00:37:46.607485+0000","last_active":"2024-04-24T00:37:48.607046+0000","last_peered":"2024-04-24T00:37:48.607046+0000","last_clean":"2024-04-24T00:37:48.607046+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:37:48.607046+0000","last_undegraded":"2024-04-24T00:37:48.607046+0000","last_fullsized":"2024-04-24T00:37:48.607046+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":932,"log_dups_size":0,"ondisk_log_size":932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:55.142994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032049069999999999,"stat_sum":{"num_bytes":44194330,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7095,"num_read_kb":32571,"num_write":1194,"num_write_kb":21148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.a","version":"232'1867","reported_seq":4114,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.619950+0000","last_change":"2024-04-24T00:37:46.609043+0000","last_active":"2024-04-24T00:37:48.619950+0000","last_peered":"2024-04-24T00:37:48.619950+0000","last_clean":"2024-04-24T00:37:48.619950+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:37:48.619950+0000","last_undegraded":"2024-04-24T00:37:48.619950+0000","last_fullsized":"2024-04-24T00:37:48.619950+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1867,"log_dups_size":0,"ondisk_log_size":1867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:47:20.613604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003056736,"stat_sum":{"num_bytes":22016457,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":9506,"num_read_kb":44890,"num_write":3222,"num_write_kb":33615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.b","version":"225'445","reported_seq":4121,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594685+0000","last_change":"2024-04-24T00:37:46.603997+0000","last_active":"2024-04-24T00:37:49.594685+0000","last_peered":"2024-04-24T00:37:49.594685+0000","last_clean":"2024-04-24T00:37:49.594685+0000","last_became_active":"2024-04-24T00:37:45.607329+0000","last_became_peered":"2024-04-24T00:37:45.607329+0000","last_unstale":"2024-04-24T00:37:49.594685+0000","last_undegraded":"2024-04-24T00:37:49.594685+0000","last_fullsized":"2024-04-24T00:37:49.594685+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:18:59.283239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031718229999999998,"stat_sum":{"num_bytes":11945415,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3983,"num_read_kb":34787,"num_write":459,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.c","version":"225'754","reported_seq":9750,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612425+0000","last_change":"2024-04-24T00:37:46.614718+0000","last_active":"2024-04-24T00:37:48.612425+0000","last_peered":"2024-04-24T00:37:48.612425+0000","last_clean":"2024-04-24T00:37:48.612425+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:37:48.612425+0000","last_undegraded":"2024-04-24T00:37:48.612425+0000","last_fullsized":"2024-04-24T00:37:48.612425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:39.565151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025462010000000001,"stat_sum":{"num_bytes":63125637,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10330,"num_read_kb":68570,"num_write":1103,"num_write_kb":35198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.d","version":"231'416","reported_seq":2741,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612058+0000","last_change":"2024-04-24T00:37:46.613660+0000","last_active":"2024-04-24T00:37:48.612058+0000","last_peered":"2024-04-24T00:37:48.612058+0000","last_clean":"2024-04-24T00:37:48.612058+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:37:48.612058+0000","last_undegraded":"2024-04-24T00:37:48.612058+0000","last_fullsized":"2024-04-24T00:37:48.612058+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:36.391472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015713649999999999,"stat_sum":{"num_bytes":6744510,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":546,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.e","version":"231'652","reported_seq":7653,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607751+0000","last_change":"2024-04-24T00:37:46.610598+0000","last_active":"2024-04-24T00:37:48.607751+0000","last_peered":"2024-04-24T00:37:48.607751+0000","last_clean":"2024-04-24T00:37:48.607751+0000","last_became_active":"2024-04-24T00:35:29.560215+0000","last_became_peered":"2024-04-24T00:35:29.560215+0000","last_unstale":"2024-04-24T00:37:48.607751+0000","last_undegraded":"2024-04-24T00:37:48.607751+0000","last_fullsized":"2024-04-24T00:37:48.607751+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:53.583917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046475290000000001,"stat_sum":{"num_bytes":31320762,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":934,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":2728103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.f","version":"231'841","reported_seq":5446,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.622810+0000","last_change":"2024-04-24T00:37:48.623765+0000","last_active":"2024-04-24T00:37:48.622810+0000","last_peered":"2024-04-24T00:37:48.622810+0000","last_clean":"2024-04-24T00:37:48.622810+0000","last_became_active":"2024-04-24T00:37:48.616654+0000","last_became_peered":"2024-04-24T00:37:48.616654+0000","last_unstale":"2024-04-24T00:37:48.622810+0000","last_undegraded":"2024-04-24T00:37:48.622810+0000","last_fullsized":"2024-04-24T00:37:48.622810+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:39:58.213721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037740320000000001,"stat_sum":{"num_bytes":27430936,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5510,"num_read_kb":46049,"num_write":1340,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":26495407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.10","version":"231'559","reported_seq":6095,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.620915+0000","last_change":"2024-04-24T00:37:46.615985+0000","last_active":"2024-04-24T00:37:48.620915+0000","last_peered":"2024-04-24T00:37:48.620915+0000","last_clean":"2024-04-24T00:37:48.620915+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:37:48.620915+0000","last_undegraded":"2024-04-24T00:37:48.620915+0000","last_fullsized":"2024-04-24T00:37:48.620915+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:24:18.385907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001356445,"stat_sum":{"num_bytes":10463442,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5981,"num_read_kb":30449,"num_write":663,"num_write_kb":20575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.11","version":"233'540","reported_seq":7929,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.606854+0000","last_change":"2024-04-24T00:37:46.609789+0000","last_active":"2024-04-24T00:37:48.606854+0000","last_peered":"2024-04-24T00:37:48.606854+0000","last_clean":"2024-04-24T00:37:48.606854+0000","last_became_active":"2024-04-24T00:35:28.722568+0000","last_became_peered":"2024-04-24T00:35:28.722568+0000","last_unstale":"2024-04-24T00:37:48.606854+0000","last_undegraded":"2024-04-24T00:37:48.606854+0000","last_fullsized":"2024-04-24T00:37:48.606854+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:25.198938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053693430000000004,"stat_sum":{"num_bytes":33084929,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7209,"num_read_kb":84250,"num_write":642,"num_write_kb":47836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":1618865,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.12","version":"224'1450","reported_seq":5482,"reported_epoch":237,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:49.595079+0000","last_change":"2024-04-24T00:37:49.595079+0000","last_active":"2024-04-24T00:37:49.595079+0000","last_peered":"2024-04-24T00:37:47.396304+0000","last_clean":"2024-04-24T00:37:40.621256+0000","last_became_active":"2024-04-24T00:37:43.717521+0000","last_became_peered":"2024-04-24T00:37:43.717521+0000","last_unstale":"2024-04-24T00:37:49.595079+0000","last_undegraded":"2024-04-24T00:37:49.595079+0000","last_fullsized":"2024-04-24T00:37:49.595079+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1450,"log_dups_size":0,"ondisk_log_size":1450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":41640823,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7403,"num_read_kb":53792,"num_write":2236,"num_write_kb":43767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":46534159,"num_keys_recovered":76,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.13","version":"232'638","reported_seq":4683,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607471+0000","last_change":"2024-04-24T00:37:46.612592+0000","last_active":"2024-04-24T00:37:48.607471+0000","last_peered":"2024-04-24T00:37:48.607471+0000","last_clean":"2024-04-24T00:37:48.607471+0000","last_became_active":"2024-04-24T00:35:28.722941+0000","last_became_peered":"2024-04-24T00:35:28.722941+0000","last_unstale":"2024-04-24T00:37:48.607471+0000","last_undegraded":"2024-04-24T00:37:48.607471+0000","last_fullsized":"2024-04-24T00:37:48.607471+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:28:06.433639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060203920000000003,"stat_sum":{"num_bytes":22578363,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4875,"num_read_kb":39170,"num_write":915,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":962033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":902529949,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":181093,"num_read_kb":1541647,"num_write":29972,"num_write_kb":1017889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4488,"num_bytes_recovered":318957047,"num_keys_recovered":439,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":31563776,"data_stored":239779518,"data_compressed":9905415,"data_compressed_allocated":24584192,"data_compressed_original":236150073,"omap_allocated":4834,"internal_metadata":0},"log_size":21432,"ondisk_log_size":21432,"up":64,"acting":57,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":34064,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574903,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80884,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290956,"statfs":{"total":96636764160,"available":96553938944,"internally_reserved":0,"allocated":5414912,"data_stored":19800134,"data_compressed":700950,"data_compressed_allocated":1851392,"data_compressed_original":18023341,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76380,"kb_used_data":9104,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94295460,"statfs":{"total":96636764160,"available":96558551040,"internally_reserved":0,"allocated":9322496,"data_stored":50932895,"data_compressed":2062361,"data_compressed_allocated":5099520,"data_compressed_original":48818538,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84400,"kb_used_data":8804,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287440,"statfs":{"total":96636764160,"available":96550338560,"internally_reserved":0,"allocated":9015296,"data_stored":53950367,"data_compressed":1619699,"data_compressed_allocated":4845568,"data_compressed_original":51871456,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74440,"kb_used_data":7164,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297400,"statfs":{"total":96636764160,"available":96560537600,"internally_reserved":0,"allocated":7335936,"data_stored":32323445,"data_compressed":1603047,"data_compressed_allocated":3485696,"data_compressed_original":30392448,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58936,"kb_used_data":5544,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312904,"statfs":{"total":96636764160,"available":96576413696,"internally_reserved":0,"allocated":5677056,"data_stored":22925582,"data_compressed":890449,"data_compressed_allocated":2191360,"data_compressed_original":21176320,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34864,"kb_used_data":3676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336976,"statfs":{"total":96636764160,"available":96601063424,"internally_reserved":0,"allocated":3764224,"data_stored":13134948,"data_compressed":119396,"data_compressed_allocated":843776,"data_compressed_original":11685888,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893152,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35332,"kb_used_data":7600,"kb_used_omap":6,"kb_used_meta":27705,"kb_avail":94336508,"statfs":{"total":96636764160,"available":96600584192,"internally_reserved":0,"allocated":7782400,"data_stored":37064126,"data_compressed":1713150,"data_compressed_allocated":3862528,"data_compressed_original":35086336,"omap_allocated":6429,"internal_metadata":28370659},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4882432,"data_stored":35057512,"data_compressed":1706165,"data_compressed_allocated":3825664,"data_compressed_original":34496512,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":917504,"data_stored":11725087,"data_compressed":119396,"data_compressed_allocated":843776,"data_compressed_original":11685888,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2830336,"data_stored":21515721,"data_compressed":890449,"data_compressed_allocated":2191360,"data_compressed_original":21176320,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4489216,"data_stored":30913584,"data_compressed":1603047,"data_compressed_allocated":3485696,"data_compressed_original":30392448,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6168576,"data_stored":52540506,"data_compressed":1619699,"data_compressed_allocated":4845568,"data_compressed_original":51871456,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":45276737,"data_compressed":1986180,"data_compressed_allocated":4759552,"data_compressed_original":44611946,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2527232,"data_stored":17799905,"data_compressed":693965,"data_compressed_allocated":1814528,"data_compressed_original":17433517,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:53.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:37:53.996 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:53.994+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:54.032 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 220.81791791699999, 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:37:54.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:37:54.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:37:54.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:54.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 220.817567002, 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 220.810678682, 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:54.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.55239705599999, 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:37:54.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.54212064000001, 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:54.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:37:54.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:54.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.080169837, 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.05413581600001, 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:37:54.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:37:54.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.840192909, 2024-04-24T00:37:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:37:54.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:37:54.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.83018096500001, 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:37:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.556802103, 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:37:54.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.081 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.54013365399999, 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.085 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.354235871, 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:37:54.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.670477514, 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:37:54.091 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:37:54.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.64826051599999, 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:37:54.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.343690678000002, 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:54.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:37:54.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:37:54.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.331065718000005, 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:54.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:37:54.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.981574231000003, 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:37:54.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.116 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.966556036, 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:54.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:37:54.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:37:54.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.073739633000002, 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:37:54.120 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:37:54.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:37:54.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:37:54.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:37:54.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.036387605999998, 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:37:54.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.130 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.130 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.130 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:54.130 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:54.130 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:54.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:54.132 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:54.126+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:54.232 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:54.230+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:54.262 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:54.263 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:54.263 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:54.263 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:54.272 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:37:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:37:54.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:54.290 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-04-24T00:37:53.713602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":903120317,"num_objects":539,"num_object_clones":0,"num_object_copies":1078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":181209,"num_read_kb":1541747,"num_write":30131,"num_write_kb":1020305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4494,"num_bytes_recovered":320728151,"num_keys_recovered":439,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21549,"ondisk_log_size":21549,"up":82,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":91,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":445236,"kb_used_data":47180,"kb_used_omap":15,"kb_used_meta":397936,"kb_avail":660157644,"statfs":{"total":676457349120,"available":676001427456,"internally_reserved":0,"allocated":48312320,"data_stored":230131497,"data_compressed":8709052,"data_compressed_allocated":22179840,"data_compressed_original":217054327,"omap_allocated":15996,"internal_metadata":407486852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001366"},"pg_stats":[{"pgid":"3.1f","version":"231'831","reported_seq":5674,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594994+0000","last_change":"2024-04-24T00:37:47.623538+0000","last_active":"2024-04-24T00:37:49.594994+0000","last_peered":"2024-04-24T00:37:49.594994+0000","last_clean":"2024-04-24T00:37:49.594994+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:37:49.594994+0000","last_undegraded":"2024-04-24T00:37:49.594994+0000","last_fullsized":"2024-04-24T00:37:49.594994+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:01:23.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019480669999999999,"stat_sum":{"num_bytes":52367280,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6142,"num_read_kb":55851,"num_write":1308,"num_write_kb":38951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1e","version":"231'977","reported_seq":5291,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.620237+0000","last_change":"2024-04-24T00:37:46.609936+0000","last_active":"2024-04-24T00:37:48.620237+0000","last_peered":"2024-04-24T00:37:48.620237+0000","last_clean":"2024-04-24T00:37:48.620237+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:37:48.620237+0000","last_undegraded":"2024-04-24T00:37:48.620237+0000","last_fullsized":"2024-04-24T00:37:48.620237+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:42.516498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003187456,"stat_sum":{"num_bytes":21004221,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7576,"num_read_kb":34899,"num_write":1269,"num_write_kb":26860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1d","version":"232'552","reported_seq":6735,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612940+0000","last_change":"2024-04-24T00:37:46.614489+0000","last_active":"2024-04-24T00:37:48.612940+0000","last_peered":"2024-04-24T00:37:48.612940+0000","last_clean":"2024-04-24T00:37:48.612940+0000","last_became_active":"2024-04-24T00:34:28.726196+0000","last_became_peered":"2024-04-24T00:34:28.726196+0000","last_unstale":"2024-04-24T00:37:48.612940+0000","last_undegraded":"2024-04-24T00:37:48.612940+0000","last_fullsized":"2024-04-24T00:37:48.612940+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:00:13.573517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001963238,"stat_sum":{"num_bytes":32555141,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7132,"num_read_kb":66702,"num_write":793,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":458933,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1c","version":"235'712","reported_seq":5393,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.611973+0000","last_change":"2024-04-24T00:37:46.613357+0000","last_active":"2024-04-24T00:37:48.611973+0000","last_peered":"2024-04-24T00:37:48.611973+0000","last_clean":"2024-04-24T00:37:48.611973+0000","last_became_active":"2024-04-24T00:34:31.747052+0000","last_became_peered":"2024-04-24T00:34:31.747052+0000","last_unstale":"2024-04-24T00:37:48.611973+0000","last_undegraded":"2024-04-24T00:37:48.611973+0000","last_fullsized":"2024-04-24T00:37:48.611973+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:11.238051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001537114,"stat_sum":{"num_bytes":32968971,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4941,"num_read_kb":61078,"num_write":909,"num_write_kb":35121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":949407,"num_keys_recovered":27,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1b","version":"231'455","reported_seq":4234,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612191+0000","last_change":"2024-04-24T00:37:46.611866+0000","last_active":"2024-04-24T00:37:48.612191+0000","last_peered":"2024-04-24T00:37:48.612191+0000","last_clean":"2024-04-24T00:37:48.612191+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:37:48.612191+0000","last_undegraded":"2024-04-24T00:37:48.612191+0000","last_fullsized":"2024-04-24T00:37:48.612191+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:08.897008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056302700000000002,"stat_sum":{"num_bytes":26704680,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3990,"num_read_kb":45455,"num_write":567,"num_write_kb":27383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1a","version":"232'374","reported_seq":5487,"reported_epoch":235,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:47.611481+0000","last_change":"2024-04-24T00:37:48.608289+0000","last_active":"2024-04-24T00:37:47.611481+0000","last_peered":"2024-04-24T00:37:46.679778+0000","last_clean":"2024-04-24T00:37:46.679778+0000","last_became_active":"2024-04-24T00:35:47.582331+0000","last_became_peered":"2024-04-24T00:35:47.582331+0000","last_unstale":"2024-04-24T00:37:47.611481+0000","last_undegraded":"2024-04-24T00:37:47.611481+0000","last_fullsized":"2024-04-24T00:37:47.611481+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048149600000000001,"stat_sum":{"num_bytes":9349759,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4387,"num_read_kb":37212,"num_write":485,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":1449843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.19","version":"228'680","reported_seq":5653,"reported_epoch":234,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:46.604822+0000","last_change":"2024-04-24T00:37:48.613702+0000","last_active":"2024-04-24T00:37:48.613702+0000","last_peered":"2024-04-24T00:37:46.604822+0000","last_clean":"2024-04-24T00:37:46.604822+0000","last_became_active":"2024-04-24T00:35:47.589674+0000","last_became_peered":"2024-04-24T00:35:47.589674+0000","last_unstale":"2024-04-24T00:37:46.604822+0000","last_undegraded":"2024-04-24T00:37:46.604822+0000","last_fullsized":"2024-04-24T00:37:46.604822+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000930561,"stat_sum":{"num_bytes":24604173,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5924,"num_read_kb":40380,"num_write":975,"num_write_kb":29815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":868041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":1,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.18","version":"235'537","reported_seq":4709,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594755+0000","last_change":"2024-04-24T00:37:46.611758+0000","last_active":"2024-04-24T00:37:49.594755+0000","last_peered":"2024-04-24T00:37:49.594755+0000","last_clean":"2024-04-24T00:37:49.594755+0000","last_became_active":"2024-04-24T00:37:46.608136+0000","last_became_peered":"2024-04-24T00:37:46.608136+0000","last_unstale":"2024-04-24T00:37:49.594755+0000","last_undegraded":"2024-04-24T00:37:49.594755+0000","last_fullsized":"2024-04-24T00:37:49.594755+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:15.418243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014921719999999999,"stat_sum":{"num_bytes":28085793,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":584,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.17","version":"231'322","reported_seq":4593,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.596110+0000","last_change":"2024-04-24T00:37:46.618751+0000","last_active":"2024-04-24T00:37:49.596110+0000","last_peered":"2024-04-24T00:37:49.596110+0000","last_clean":"2024-04-24T00:37:49.596110+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:37:49.596110+0000","last_undegraded":"2024-04-24T00:37:49.596110+0000","last_fullsized":"2024-04-24T00:37:49.596110+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:16.564344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014710190000000001,"stat_sum":{"num_bytes":14945483,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":417,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.16","version":"231'670","reported_seq":6319,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612865+0000","last_change":"2024-04-24T00:37:46.605862+0000","last_active":"2024-04-24T00:37:48.612865+0000","last_peered":"2024-04-24T00:37:48.612865+0000","last_clean":"2024-04-24T00:37:48.612865+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:37:48.612865+0000","last_undegraded":"2024-04-24T00:37:48.612865+0000","last_fullsized":"2024-04-24T00:37:48.612865+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:28:04.964561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064798799999999995,"stat_sum":{"num_bytes":32925991,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7147,"num_read_kb":50536,"num_write":1023,"num_write_kb":38907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.15","version":"231'613","reported_seq":6812,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612606+0000","last_change":"2024-04-24T00:37:46.604886+0000","last_active":"2024-04-24T00:37:48.612606+0000","last_peered":"2024-04-24T00:37:48.612606+0000","last_clean":"2024-04-24T00:37:48.612606+0000","last_became_active":"2024-04-24T00:35:28.725126+0000","last_became_peered":"2024-04-24T00:35:28.725126+0000","last_unstale":"2024-04-24T00:37:48.612606+0000","last_undegraded":"2024-04-24T00:37:48.612606+0000","last_fullsized":"2024-04-24T00:37:48.612606+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:18:28.502951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056773200000000005,"stat_sum":{"num_bytes":36441735,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":878,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":2312722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.14","version":"228'476","reported_seq":4986,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607048+0000","last_change":"2024-04-24T00:37:46.610711+0000","last_active":"2024-04-24T00:37:48.607048+0000","last_peered":"2024-04-24T00:37:48.607048+0000","last_clean":"2024-04-24T00:37:48.607048+0000","last_became_active":"2024-04-24T00:35:28.722992+0000","last_became_peered":"2024-04-24T00:35:28.722992+0000","last_unstale":"2024-04-24T00:37:48.607048+0000","last_undegraded":"2024-04-24T00:37:48.607048+0000","last_fullsized":"2024-04-24T00:37:48.607048+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:45:17.421114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058891000000000004,"stat_sum":{"num_bytes":23091924,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4140,"num_read_kb":41580,"num_write":565,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1929866,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":449,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.619922+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:37:48.619922+0000","last_peered":"2024-04-24T00:37:48.619922+0000","last_clean":"2024-04-24T00:37:48.619922+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:37:48.619922+0000","last_undegraded":"2024-04-24T00:37:48.619922+0000","last_fullsized":"2024-04-24T00:37:48.619922+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"232'529","reported_seq":6038,"reported_epoch":237,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-24T00:37:49.599990+0000","last_change":"2024-04-24T00:37:49.631656+0000","last_active":"2024-04-24T00:37:47.612198+0000","last_peered":"2024-04-24T00:37:46.617891+0000","last_clean":"2024-04-24T00:37:46.617891+0000","last_became_active":"2024-04-24T00:37:46.611693+0000","last_became_peered":"2024-04-24T00:37:46.611693+0000","last_unstale":"2024-04-24T00:37:49.599990+0000","last_undegraded":"2024-04-24T00:37:49.599990+0000","last_fullsized":"2024-04-24T00:37:49.595246+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001201978,"stat_sum":{"num_bytes":48215459,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5114,"num_read_kb":58057,"num_write":872,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":44596023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":24}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.624832+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:37:48.624832+0000","last_peered":"2024-04-24T00:37:48.624832+0000","last_clean":"2024-04-24T00:37:48.624832+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:37:48.624832+0000","last_undegraded":"2024-04-24T00:37:48.624832+0000","last_fullsized":"2024-04-24T00:37:48.624832+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"231'580","reported_seq":4521,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.595609+0000","last_change":"2024-04-24T00:37:47.621693+0000","last_active":"2024-04-24T00:37:49.595609+0000","last_peered":"2024-04-24T00:37:49.595609+0000","last_clean":"2024-04-24T00:37:49.595609+0000","last_became_active":"2024-04-24T00:37:47.617077+0000","last_became_peered":"2024-04-24T00:37:47.617077+0000","last_unstale":"2024-04-24T00:37:49.595609+0000","last_undegraded":"2024-04-24T00:37:49.595609+0000","last_fullsized":"2024-04-24T00:37:49.595609+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:08:32.489010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002332027,"stat_sum":{"num_bytes":43812937,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4338,"num_read_kb":41656,"num_write":809,"num_write_kb":26241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.595036+0000","last_change":"2024-04-24T00:37:42.650295+0000","last_active":"2024-04-24T00:37:49.595036+0000","last_peered":"2024-04-24T00:37:49.595036+0000","last_clean":"2024-04-24T00:37:49.595036+0000","last_became_active":"2024-04-24T00:37:42.649872+0000","last_became_peered":"2024-04-24T00:37:42.649872+0000","last_unstale":"2024-04-24T00:37:49.595036+0000","last_undegraded":"2024-04-24T00:37:49.595036+0000","last_fullsized":"2024-04-24T00:37:49.595036+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:39:37.552779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"234'495","reported_seq":3892,"reported_epoch":236,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:48.613409+0000","last_change":"2024-04-24T00:37:48.613409+0000","last_active":"2024-04-24T00:37:48.613409+0000","last_peered":"2024-04-24T00:37:46.697143+0000","last_clean":"2024-04-24T00:37:46.697143+0000","last_became_active":"2024-04-24T00:35:47.586287+0000","last_became_peered":"2024-04-24T00:35:47.586287+0000","last_unstale":"2024-04-24T00:37:48.613409+0000","last_undegraded":"2024-04-24T00:37:48.613409+0000","last_fullsized":"2024-04-24T00:37:48.613409+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082295399999999998,"stat_sum":{"num_bytes":23735278,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3403,"num_read_kb":34380,"num_write":703,"num_write_kb":26463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1579365,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.624738+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:37:48.624738+0000","last_peered":"2024-04-24T00:37:48.624738+0000","last_clean":"2024-04-24T00:37:48.624738+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:37:48.624738+0000","last_undegraded":"2024-04-24T00:37:48.624738+0000","last_fullsized":"2024-04-24T00:37:48.624738+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"231'616","reported_seq":5955,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612474+0000","last_change":"2024-04-24T00:37:46.615072+0000","last_active":"2024-04-24T00:37:48.612474+0000","last_peered":"2024-04-24T00:37:48.612474+0000","last_clean":"2024-04-24T00:37:48.612474+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:37:48.612474+0000","last_undegraded":"2024-04-24T00:37:48.612474+0000","last_fullsized":"2024-04-24T00:37:48.612474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:45:18.835496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002440181,"stat_sum":{"num_bytes":34823236,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5966,"num_read_kb":55481,"num_write":794,"num_write_kb":40841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.3","version":"235'480","reported_seq":6213,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607853+0000","last_change":"2024-04-24T00:37:47.629526+0000","last_active":"2024-04-24T00:37:48.607853+0000","last_peered":"2024-04-24T00:37:48.607853+0000","last_clean":"2024-04-24T00:37:48.607853+0000","last_became_active":"2024-04-24T00:37:47.623000+0000","last_became_peered":"2024-04-24T00:37:47.623000+0000","last_unstale":"2024-04-24T00:37:48.607853+0000","last_undegraded":"2024-04-24T00:37:48.607853+0000","last_fullsized":"2024-04-24T00:37:48.607853+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:41:21.537719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026935660000000001,"stat_sum":{"num_bytes":3134459,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4940,"num_read_kb":51361,"num_write":642,"num_write_kb":30821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":29414668,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":375,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.611822+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:37:48.611822+0000","last_peered":"2024-04-24T00:37:48.611822+0000","last_clean":"2024-04-24T00:37:48.611822+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:37:48.611822+0000","last_undegraded":"2024-04-24T00:37:48.611822+0000","last_fullsized":"2024-04-24T00:37:48.611822+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"231'595","reported_seq":7557,"reported_epoch":234,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:46.603085+0000","last_change":"2024-04-24T00:37:48.614146+0000","last_active":"2024-04-24T00:37:48.614146+0000","last_peered":"2024-04-24T00:37:46.603085+0000","last_clean":"2024-04-24T00:37:46.603085+0000","last_became_active":"2024-04-24T00:35:47.586354+0000","last_became_peered":"2024-04-24T00:35:47.586354+0000","last_unstale":"2024-04-24T00:37:46.603085+0000","last_undegraded":"2024-04-24T00:37:46.603085+0000","last_fullsized":"2024-04-24T00:37:46.603085+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064418099999999996,"stat_sum":{"num_bytes":15384600,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7270,"num_read_kb":65726,"num_write":672,"num_write_kb":34854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3429938,"num_keys_recovered":22,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":1,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":444,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.614774+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:37:48.614774+0000","last_peered":"2024-04-24T00:37:48.614774+0000","last_clean":"2024-04-24T00:37:48.614774+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:37:48.614774+0000","last_undegraded":"2024-04-24T00:37:48.614774+0000","last_fullsized":"2024-04-24T00:37:48.614774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"231'426","reported_seq":4103,"reported_epoch":236,"state":"peering","last_fresh":"2024-04-24T00:37:48.610467+0000","last_change":"2024-04-24T00:37:49.596425+0000","last_active":"2024-04-24T00:37:49.596425+0000","last_peered":"2024-04-24T00:37:48.610467+0000","last_clean":"2024-04-24T00:37:48.610467+0000","last_became_active":"2024-04-24T00:37:45.607305+0000","last_became_peered":"2024-04-24T00:37:45.607305+0000","last_unstale":"2024-04-24T00:37:48.610467+0000","last_undegraded":"2024-04-24T00:37:48.610467+0000","last_fullsized":"2024-04-24T00:37:48.610467+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033054920000000002,"stat_sum":{"num_bytes":26338783,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2119,"num_read_kb":33775,"num_write":541,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":27566656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.622167+0000","last_change":"2024-04-24T00:37:48.620512+0000","last_active":"2024-04-24T00:37:48.622167+0000","last_peered":"2024-04-24T00:37:48.622167+0000","last_clean":"2024-04-24T00:37:48.622167+0000","last_became_active":"2024-04-24T00:37:48.620159+0000","last_became_peered":"2024-04-24T00:37:48.620159+0000","last_unstale":"2024-04-24T00:37:48.622167+0000","last_undegraded":"2024-04-24T00:37:48.622167+0000","last_fullsized":"2024-04-24T00:37:48.622167+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:47.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"231'557","reported_seq":2562,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612119+0000","last_change":"2024-04-24T00:37:46.613641+0000","last_active":"2024-04-24T00:37:48.612119+0000","last_peered":"2024-04-24T00:37:48.612119+0000","last_clean":"2024-04-24T00:37:48.612119+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:37:48.612119+0000","last_undegraded":"2024-04-24T00:37:48.612119+0000","last_fullsized":"2024-04-24T00:37:48.612119+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:10.421466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019991789999999998,"stat_sum":{"num_bytes":26453257,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2553,"num_read_kb":24145,"num_write":731,"num_write_kb":27938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":375,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.611902+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:37:48.611902+0000","last_peered":"2024-04-24T00:37:48.611902+0000","last_clean":"2024-04-24T00:37:48.611902+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:37:48.611902+0000","last_undegraded":"2024-04-24T00:37:48.611902+0000","last_fullsized":"2024-04-24T00:37:48.611902+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":405,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594456+0000","last_change":"2024-04-24T00:37:45.610754+0000","last_active":"2024-04-24T00:37:49.594456+0000","last_peered":"2024-04-24T00:37:49.594456+0000","last_clean":"2024-04-24T00:37:49.594456+0000","last_became_active":"2024-04-24T00:37:45.609794+0000","last_became_peered":"2024-04-24T00:37:45.609794+0000","last_unstale":"2024-04-24T00:37:49.594456+0000","last_undegraded":"2024-04-24T00:37:49.594456+0000","last_fullsized":"2024-04-24T00:37:49.594456+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:27.857798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"231'861","reported_seq":6617,"reported_epoch":234,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:46.607508+0000","last_change":"2024-04-24T00:37:48.607922+0000","last_active":"2024-04-24T00:37:48.607922+0000","last_peered":"2024-04-24T00:37:46.607508+0000","last_clean":"2024-04-24T00:37:46.607508+0000","last_became_active":"2024-04-24T00:35:54.570587+0000","last_became_peered":"2024-04-24T00:35:54.570587+0000","last_unstale":"2024-04-24T00:37:46.607508+0000","last_undegraded":"2024-04-24T00:37:46.607508+0000","last_fullsized":"2024-04-24T00:37:46.607508+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":861,"log_dups_size":0,"ondisk_log_size":861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031730159999999999,"stat_sum":{"num_bytes":31041185,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7207,"num_read_kb":68772,"num_write":1181,"num_write_kb":32528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":5566512,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":1,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.9","version":"232'932","reported_seq":4792,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607046+0000","last_change":"2024-04-24T00:37:46.607485+0000","last_active":"2024-04-24T00:37:48.607046+0000","last_peered":"2024-04-24T00:37:48.607046+0000","last_clean":"2024-04-24T00:37:48.607046+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:37:48.607046+0000","last_undegraded":"2024-04-24T00:37:48.607046+0000","last_fullsized":"2024-04-24T00:37:48.607046+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":932,"log_dups_size":0,"ondisk_log_size":932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:55.142994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032049069999999999,"stat_sum":{"num_bytes":44194330,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7095,"num_read_kb":32571,"num_write":1194,"num_write_kb":21148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.a","version":"232'1867","reported_seq":4114,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.619950+0000","last_change":"2024-04-24T00:37:46.609043+0000","last_active":"2024-04-24T00:37:48.619950+0000","last_peered":"2024-04-24T00:37:48.619950+0000","last_clean":"2024-04-24T00:37:48.619950+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:37:48.619950+0000","last_undegraded":"2024-04-24T00:37:48.619950+0000","last_fullsized":"2024-04-24T00:37:48.619950+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1867,"log_dups_size":0,"ondisk_log_size":1867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:47:20.613604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003056736,"stat_sum":{"num_bytes":22016457,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":9506,"num_read_kb":44890,"num_write":3222,"num_write_kb":33615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.b","version":"225'445","reported_seq":4121,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594685+0000","last_change":"2024-04-24T00:37:46.603997+0000","last_active":"2024-04-24T00:37:49.594685+0000","last_peered":"2024-04-24T00:37:49.594685+0000","last_clean":"2024-04-24T00:37:49.594685+0000","last_became_active":"2024-04-24T00:37:45.607329+0000","last_became_peered":"2024-04-24T00:37:45.607329+0000","last_unstale":"2024-04-24T00:37:49.594685+0000","last_undegraded":"2024-04-24T00:37:49.594685+0000","last_fullsized":"2024-04-24T00:37:49.594685+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:18:59.283239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031718229999999998,"stat_sum":{"num_bytes":11945415,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3983,"num_read_kb":34787,"num_write":459,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.c","version":"225'754","reported_seq":9750,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612425+0000","last_change":"2024-04-24T00:37:46.614718+0000","last_active":"2024-04-24T00:37:48.612425+0000","last_peered":"2024-04-24T00:37:48.612425+0000","last_clean":"2024-04-24T00:37:48.612425+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:37:48.612425+0000","last_undegraded":"2024-04-24T00:37:48.612425+0000","last_fullsized":"2024-04-24T00:37:48.612425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:39.565151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025462010000000001,"stat_sum":{"num_bytes":63125637,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10330,"num_read_kb":68570,"num_write":1103,"num_write_kb":35198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.d","version":"231'416","reported_seq":2741,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.612058+0000","last_change":"2024-04-24T00:37:46.613660+0000","last_active":"2024-04-24T00:37:48.612058+0000","last_peered":"2024-04-24T00:37:48.612058+0000","last_clean":"2024-04-24T00:37:48.612058+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:37:48.612058+0000","last_undegraded":"2024-04-24T00:37:48.612058+0000","last_fullsized":"2024-04-24T00:37:48.612058+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:36.391472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015713649999999999,"stat_sum":{"num_bytes":6744510,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":546,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.e","version":"231'652","reported_seq":7653,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607751+0000","last_change":"2024-04-24T00:37:46.610598+0000","last_active":"2024-04-24T00:37:48.607751+0000","last_peered":"2024-04-24T00:37:48.607751+0000","last_clean":"2024-04-24T00:37:48.607751+0000","last_became_active":"2024-04-24T00:35:29.560215+0000","last_became_peered":"2024-04-24T00:35:29.560215+0000","last_unstale":"2024-04-24T00:37:48.607751+0000","last_undegraded":"2024-04-24T00:37:48.607751+0000","last_fullsized":"2024-04-24T00:37:48.607751+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:53.583917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046475290000000001,"stat_sum":{"num_bytes":31320762,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":934,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":2728103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.f","version":"231'841","reported_seq":5446,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.622810+0000","last_change":"2024-04-24T00:37:48.623765+0000","last_active":"2024-04-24T00:37:48.622810+0000","last_peered":"2024-04-24T00:37:48.622810+0000","last_clean":"2024-04-24T00:37:48.622810+0000","last_became_active":"2024-04-24T00:37:48.616654+0000","last_became_peered":"2024-04-24T00:37:48.616654+0000","last_unstale":"2024-04-24T00:37:48.622810+0000","last_undegraded":"2024-04-24T00:37:48.622810+0000","last_fullsized":"2024-04-24T00:37:48.622810+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:39:58.213721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037740320000000001,"stat_sum":{"num_bytes":27430936,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5510,"num_read_kb":46049,"num_write":1340,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":26495407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.10","version":"231'559","reported_seq":6095,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.620915+0000","last_change":"2024-04-24T00:37:46.615985+0000","last_active":"2024-04-24T00:37:48.620915+0000","last_peered":"2024-04-24T00:37:48.620915+0000","last_clean":"2024-04-24T00:37:48.620915+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:37:48.620915+0000","last_undegraded":"2024-04-24T00:37:48.620915+0000","last_fullsized":"2024-04-24T00:37:48.620915+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:24:18.385907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001356445,"stat_sum":{"num_bytes":10463442,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5981,"num_read_kb":30449,"num_write":663,"num_write_kb":20575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.11","version":"233'540","reported_seq":7929,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.606854+0000","last_change":"2024-04-24T00:37:46.609789+0000","last_active":"2024-04-24T00:37:48.606854+0000","last_peered":"2024-04-24T00:37:48.606854+0000","last_clean":"2024-04-24T00:37:48.606854+0000","last_became_active":"2024-04-24T00:35:28.722568+0000","last_became_peered":"2024-04-24T00:35:28.722568+0000","last_unstale":"2024-04-24T00:37:48.606854+0000","last_undegraded":"2024-04-24T00:37:48.606854+0000","last_fullsized":"2024-04-24T00:37:48.606854+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:25.198938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053693430000000004,"stat_sum":{"num_bytes":33084929,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7209,"num_read_kb":84250,"num_write":642,"num_write_kb":47836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":1618865,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.12","version":"224'1450","reported_seq":5482,"reported_epoch":237,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:49.595079+0000","last_change":"2024-04-24T00:37:49.595079+0000","last_active":"2024-04-24T00:37:49.595079+0000","last_peered":"2024-04-24T00:37:47.396304+0000","last_clean":"2024-04-24T00:37:40.621256+0000","last_became_active":"2024-04-24T00:37:43.717521+0000","last_became_peered":"2024-04-24T00:37:43.717521+0000","last_unstale":"2024-04-24T00:37:49.595079+0000","last_undegraded":"2024-04-24T00:37:49.595079+0000","last_fullsized":"2024-04-24T00:37:49.595079+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1450,"log_dups_size":0,"ondisk_log_size":1450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":41640823,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7403,"num_read_kb":53792,"num_write":2236,"num_write_kb":43767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":46534159,"num_keys_recovered":76,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.13","version":"232'638","reported_seq":4683,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-24T00:37:48.607471+0000","last_change":"2024-04-24T00:37:46.612592+0000","last_active":"2024-04-24T00:37:48.607471+0000","last_peered":"2024-04-24T00:37:48.607471+0000","last_clean":"2024-04-24T00:37:48.607471+0000","last_became_active":"2024-04-24T00:35:28.722941+0000","last_became_peered":"2024-04-24T00:35:28.722941+0000","last_unstale":"2024-04-24T00:37:48.607471+0000","last_undegraded":"2024-04-24T00:37:48.607471+0000","last_fullsized":"2024-04-24T00:37:48.607471+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:28:06.433639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060203920000000003,"stat_sum":{"num_bytes":22578363,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4875,"num_read_kb":39170,"num_write":915,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":962033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":902529949,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":181093,"num_read_kb":1541647,"num_write":29972,"num_write_kb":1017889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4488,"num_bytes_recovered":318957047,"num_keys_recovered":439,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":31563776,"data_stored":239779518,"data_compressed":9905415,"data_compressed_allocated":24584192,"data_compressed_original":236150073,"omap_allocated":4834,"internal_metadata":0},"log_size":21432,"ondisk_log_size":21432,"up":64,"acting":57,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":34064,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574903,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80884,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290956,"statfs":{"total":96636764160,"available":96553938944,"internally_reserved":0,"allocated":5414912,"data_stored":19800134,"data_compressed":700950,"data_compressed_allocated":1851392,"data_compressed_original":18023341,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76380,"kb_used_data":9104,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94295460,"statfs":{"total":96636764160,"available":96558551040,"internally_reserved":0,"allocated":9322496,"data_stored":50932895,"data_compressed":2062361,"data_compressed_allocated":5099520,"data_compressed_original":48818538,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84400,"kb_used_data":8804,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94287440,"statfs":{"total":96636764160,"available":96550338560,"internally_reserved":0,"allocated":9015296,"data_stored":53950367,"data_compressed":1619699,"data_compressed_allocated":4845568,"data_compressed_original":51871456,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74440,"kb_used_data":7164,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297400,"statfs":{"total":96636764160,"available":96560537600,"internally_reserved":0,"allocated":7335936,"data_stored":32323445,"data_compressed":1603047,"data_compressed_allocated":3485696,"data_compressed_original":30392448,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58936,"kb_used_data":5544,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94312904,"statfs":{"total":96636764160,"available":96576413696,"internally_reserved":0,"allocated":5677056,"data_stored":22925582,"data_compressed":890449,"data_compressed_allocated":2191360,"data_compressed_original":21176320,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34864,"kb_used_data":3676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336976,"statfs":{"total":96636764160,"available":96601063424,"internally_reserved":0,"allocated":3764224,"data_stored":13134948,"data_compressed":119396,"data_compressed_allocated":843776,"data_compressed_original":11685888,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893152,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35332,"kb_used_data":7600,"kb_used_omap":6,"kb_used_meta":27705,"kb_avail":94336508,"statfs":{"total":96636764160,"available":96600584192,"internally_reserved":0,"allocated":7782400,"data_stored":37064126,"data_compressed":1713150,"data_compressed_allocated":3862528,"data_compressed_original":35086336,"omap_allocated":6429,"internal_metadata":28370659},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4882432,"data_stored":35057512,"data_compressed":1706165,"data_compressed_allocated":3825664,"data_compressed_original":34496512,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":917504,"data_stored":11725087,"data_compressed":119396,"data_compressed_allocated":843776,"data_compressed_original":11685888,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2830336,"data_stored":21515721,"data_compressed":890449,"data_compressed_allocated":2191360,"data_compressed_original":21176320,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4489216,"data_stored":30913584,"data_compressed":1603047,"data_compressed_allocated":3485696,"data_compressed_original":30392448,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6168576,"data_stored":52540506,"data_compressed":1619699,"data_compressed_allocated":4845568,"data_compressed_original":51871456,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":45276737,"data_compressed":1986180,"data_compressed_allocated":4759552,"data_compressed_original":44611946,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2527232,"data_stored":17799905,"data_compressed":693965,"data_compressed_allocated":1814528,"data_compressed_original":17433517,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:54.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-24T00:37:54.337 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:54.334+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:54.434 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:54.430+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:54.444 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:54.444 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:54.444 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:54.445 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:54.445 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:54.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:54.544 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:54.542+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:37:54.635 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:54.635 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:54.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.41709508100001, 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:37:54.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:37:54.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:37:54.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:37:54.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:37:54.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:37:54.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:37:54.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:37:54.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:37:54.642 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.642 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.642 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.642 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:54.642 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 206.99530938699999, 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:37:54.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.85318182099999, 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.648 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:37:54.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:37:54.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:37:54.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:54.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:54.655 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.71541003900001, 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:37:54.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:37:54.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:54.661 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:54.662 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 140.677927721, 2024-04-24T00:37:54.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:37:54.662 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.662 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.662 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.663 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:37:54.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.663 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:54.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:54.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:37:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:37:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:37:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:37:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:37:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:37:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:37:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:37:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:37:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:37:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:37:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:37:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:37:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:37:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:37:54.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:37:54.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:37:54.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:37:54.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.672 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 137.015581857, 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.673 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:54.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:37:54.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:37:54.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:37:54.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 137.005723821, 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.679 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:37:54.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:37:54.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.686 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.686 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.686 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:54.686 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 136.985374213, 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.687 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:54.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:37:54.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:37:54.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:37:54.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:37:54.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.690 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.690 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:37:54.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:37:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:37:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:37:54.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:37:54.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.695 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.695 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.695 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:37:54.695 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:37:54.695 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.696 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.696 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.696 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.696 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.696 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:37:54.696 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:54.696 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.951328878, 2024-04-24T00:37:54.696 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.697 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.698 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.699 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.700 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.949019042, 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.701 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:37:54.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:37:54.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.00130115, 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:54.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:37:54.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:54.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:54.713 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.111911154, 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:54.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:37:54.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.068435357, 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:37:54.719 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.720 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.721 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.09446205499999, 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.723 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.728 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.116032598999993, 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:37:54.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.734 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 95.082030176999993, 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157093 3.13 3:cd0b7234:::rbd_data.115ee709dad1.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.745424941000003, 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12167272799999999, 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157093 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.739 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885996+0000", 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.426e-06 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886003+0000", 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8859999999999998e-06 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886020+0000", 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6135000000000001e-05 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886027+0000", 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3379999999999999e-06 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886099+0000", 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1802000000000001e-05 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886260+0000", 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016156800000000001 2024-04-24T00:37:54.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.895327+0000", 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090669270000000007 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007496+0000", 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112168567 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007520+0000", 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4646000000000001e-05 2024-04-24T00:37:54.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007599+0000", 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8424999999999995e-05 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007668+0000", 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9008000000000001e-05 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:157093 3.13 e197/106 ondisk, result = 0)", 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.714006902999998, 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090093925000000005, 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917416+0000", 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917426+0000", 2024-04-24T00:37:54.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917465+0000", 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9295999999999998e-05 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917484+0000", 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007420+0000", 2024-04-24T00:37:54.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089936099000000005 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007435+0000", 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4966999999999999e-05 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007507+0000", 2024-04-24T00:37:54.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2206000000000006e-05 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169765 3.13 3:c9e19d2e:::rbd_data.115ee24d4bb.0000000000000011:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.771641303000003, 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096474246999999999, 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.749 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169765 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:54.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859781+0000", 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859792+0000", 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859819+0000", 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7025999999999998e-05 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859833+0000", 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3763e-05 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859956+0000", 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122989 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.860147+0000", 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190843 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.866555+0000", 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064084540000000001 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.955923+0000", 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089367567999999994 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956056+0000", 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013292600000000001 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956157+0000", 2024-04-24T00:37:54.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101029 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956253+0000", 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5921999999999994e-05 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:173908 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [delete] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.962562031, 2024-04-24T00:37:54.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97469691999999997, 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 173908 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668860+0000", 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7269999999999999e-06 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668868+0000", 2024-04-24T00:37:54.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3799999999999994e-06 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668889+0000", 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0619999999999999e-05 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668906+0000", 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7303e-05 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669040+0000", 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134213 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669057+0000", 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7045999999999999e-05 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636000+0000", 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96694319299999998 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636222+0000", 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000221411 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636415+0000", 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193048 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.638630+0000", 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002215276 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.640380+0000", 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017497420000000001 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643297+0000", 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029172339999999999 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643383+0000", 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5566e-05 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:54.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:54.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643555+0000", 2024-04-24T00:37:54.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017216099999999999 2024-04-24T00:37:54.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.762 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:54.762 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:54.762 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:54.763 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:54.630+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:54.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:54.765 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:37:54.765 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 109/1078 objects degraded (10.111%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":269,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":242,"num_osds":8,"num_up_osds":8,"osd_up_since":1713918914,"num_in_osds":7,"osd_in_since":1713919061,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":33},{"state_name":"remapped+peering","count":6},{"state_name":"peering","count":1},{"state_name":"activating+undersized+degraded+remapped","count":1}],"num_pgs":41,"num_pools":3,"num_objects":539,"data_bytes":903120317,"bytes_used":455921664,"bytes_avail":676001427456,"bytes_total":676457349120,"inactive_pgs_ratio":0.19512194395065308,"degraded_objects":109,"degraded_total":1078,"degraded_ratio":0.10111317254174397,"recovering_objects_per_sec":250,"recovering_bytes_per_sec":7656196,"recovering_keys_per_sec":10,"num_objects_recovered":1002,"num_bytes_recovered":30635685,"num_keys_recovered":44,"read_bytes_sec":100828,"write_bytes_sec":7677,"read_op_per_sec":114,"write_op_per_sec":28},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-24T00:36:15.687147+0000","services":{}},"progress_events":{}} 2024-04-24T00:37:54.765 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-04-24T00:37:54.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:37:54.809 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:54.806+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:54.866 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:54.862+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:54.936 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:54.936 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:54.936 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:54.937 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:54.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:54.967 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:54.962+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:55.068 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:55.066+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:55.076 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:55.076 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:55.076 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:55.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:55.116 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:37:55.117 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:37:55.135 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-04-24T00:37:54.597477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":862758985,"num_objects":519,"num_object_clones":0,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":181341,"num_read_kb":1541857,"num_write":30168,"num_write_kb":1020315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5080,"num_bytes_recovered":328505498,"num_keys_recovered":482,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21587,"ondisk_log_size":21587,"up":82,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":464132,"kb_used_data":43840,"kb_used_omap":15,"kb_used_meta":420144,"kb_avail":660138748,"statfs":{"total":676457349120,"available":675982077952,"internally_reserved":0,"allocated":44892160,"data_stored":174178964,"data_compressed":8363483,"data_compressed_allocated":18411520,"data_compressed_original":160946595,"omap_allocated":15996,"internal_metadata":430227844},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-40361332,"num_objects":-20,"num_object_clones":0,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-20,"num_whiteouts":0,"num_read":132,"num_read_kb":110,"num_write":37,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":7777347,"num_keys_recovered":43,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.002118"},"pg_stats":[{"pgid":"3.1f","version":"231'831","reported_seq":5674,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594994+0000","last_change":"2024-04-24T00:37:47.623538+0000","last_active":"2024-04-24T00:37:49.594994+0000","last_peered":"2024-04-24T00:37:49.594994+0000","last_clean":"2024-04-24T00:37:49.594994+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:37:49.594994+0000","last_undegraded":"2024-04-24T00:37:49.594994+0000","last_fullsized":"2024-04-24T00:37:49.594994+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:01:23.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019480669999999999,"stat_sum":{"num_bytes":52367280,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6142,"num_read_kb":55851,"num_write":1308,"num_write_kb":38951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1e","version":"231'977","reported_seq":5301,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.641642+0000","last_change":"2024-04-24T00:37:46.609936+0000","last_active":"2024-04-24T00:37:53.641642+0000","last_peered":"2024-04-24T00:37:53.641642+0000","last_clean":"2024-04-24T00:37:53.641642+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:37:53.641642+0000","last_undegraded":"2024-04-24T00:37:53.641642+0000","last_fullsized":"2024-04-24T00:37:53.641642+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:42.516498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003187456,"stat_sum":{"num_bytes":21004221,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7576,"num_read_kb":34899,"num_write":1269,"num_write_kb":26860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1d","version":"232'552","reported_seq":6745,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.609139+0000","last_change":"2024-04-24T00:37:46.614489+0000","last_active":"2024-04-24T00:37:53.609139+0000","last_peered":"2024-04-24T00:37:53.609139+0000","last_clean":"2024-04-24T00:37:53.609139+0000","last_became_active":"2024-04-24T00:34:28.726196+0000","last_became_peered":"2024-04-24T00:34:28.726196+0000","last_unstale":"2024-04-24T00:37:53.609139+0000","last_undegraded":"2024-04-24T00:37:53.609139+0000","last_fullsized":"2024-04-24T00:37:53.609139+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:00:13.573517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001963238,"stat_sum":{"num_bytes":32555141,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7132,"num_read_kb":66702,"num_write":793,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":458933,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1c","version":"240'718","reported_seq":5410,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608467+0000","last_change":"2024-04-24T00:37:46.613357+0000","last_active":"2024-04-24T00:37:53.608467+0000","last_peered":"2024-04-24T00:37:53.608467+0000","last_clean":"2024-04-24T00:37:53.608467+0000","last_became_active":"2024-04-24T00:34:31.747052+0000","last_became_peered":"2024-04-24T00:34:31.747052+0000","last_unstale":"2024-04-24T00:37:53.608467+0000","last_undegraded":"2024-04-24T00:37:53.608467+0000","last_fullsized":"2024-04-24T00:37:53.608467+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:11.238051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001537114,"stat_sum":{"num_bytes":31150299,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":4957,"num_read_kb":61091,"num_write":917,"num_write_kb":35125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":949407,"num_keys_recovered":27,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1b","version":"231'455","reported_seq":4244,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608451+0000","last_change":"2024-04-24T00:37:46.611866+0000","last_active":"2024-04-24T00:37:53.608451+0000","last_peered":"2024-04-24T00:37:53.608451+0000","last_clean":"2024-04-24T00:37:53.608451+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:37:53.608451+0000","last_undegraded":"2024-04-24T00:37:53.608451+0000","last_fullsized":"2024-04-24T00:37:53.608451+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:08.897008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056302700000000002,"stat_sum":{"num_bytes":26704680,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3990,"num_read_kb":45455,"num_write":567,"num_write_kb":27383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1a","version":"232'374","reported_seq":5743,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606727+0000","last_change":"2024-04-24T00:37:51.619432+0000","last_active":"2024-04-24T00:37:53.606727+0000","last_peered":"2024-04-24T00:37:53.606727+0000","last_clean":"2024-04-24T00:37:53.606727+0000","last_became_active":"2024-04-24T00:37:51.618422+0000","last_became_peered":"2024-04-24T00:37:51.618422+0000","last_unstale":"2024-04-24T00:37:53.606727+0000","last_undegraded":"2024-04-24T00:37:53.606727+0000","last_fullsized":"2024-04-24T00:37:53.606727+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:29:27.578800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048149600000000001,"stat_sum":{"num_bytes":9349759,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4390,"num_read_kb":37215,"num_write":485,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2899650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.19","version":"237'681","reported_seq":5676,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606240+0000","last_change":"2024-04-24T00:37:52.617102+0000","last_active":"2024-04-24T00:37:53.606240+0000","last_peered":"2024-04-24T00:37:53.606240+0000","last_clean":"2024-04-24T00:37:53.606240+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:37:53.606240+0000","last_undegraded":"2024-04-24T00:37:53.606240+0000","last_fullsized":"2024-04-24T00:37:53.606240+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:05:48.471540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000930561,"stat_sum":{"num_bytes":22788004,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5927,"num_read_kb":40381,"num_write":976,"num_write_kb":29815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.18","version":"235'537","reported_seq":4709,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594755+0000","last_change":"2024-04-24T00:37:46.611758+0000","last_active":"2024-04-24T00:37:49.594755+0000","last_peered":"2024-04-24T00:37:49.594755+0000","last_clean":"2024-04-24T00:37:49.594755+0000","last_became_active":"2024-04-24T00:37:46.608136+0000","last_became_peered":"2024-04-24T00:37:46.608136+0000","last_unstale":"2024-04-24T00:37:49.594755+0000","last_undegraded":"2024-04-24T00:37:49.594755+0000","last_fullsized":"2024-04-24T00:37:49.594755+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:15.418243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014921719999999999,"stat_sum":{"num_bytes":28085793,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":584,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.17","version":"231'322","reported_seq":4593,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.596110+0000","last_change":"2024-04-24T00:37:46.618751+0000","last_active":"2024-04-24T00:37:49.596110+0000","last_peered":"2024-04-24T00:37:49.596110+0000","last_clean":"2024-04-24T00:37:49.596110+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:37:49.596110+0000","last_undegraded":"2024-04-24T00:37:49.596110+0000","last_fullsized":"2024-04-24T00:37:49.596110+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:16.564344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014710190000000001,"stat_sum":{"num_bytes":14945483,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":417,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.16","version":"231'670","reported_seq":6329,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607040+0000","last_change":"2024-04-24T00:37:46.605862+0000","last_active":"2024-04-24T00:37:53.607040+0000","last_peered":"2024-04-24T00:37:53.607040+0000","last_clean":"2024-04-24T00:37:53.607040+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:37:53.607040+0000","last_undegraded":"2024-04-24T00:37:53.607040+0000","last_fullsized":"2024-04-24T00:37:53.607040+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:28:04.964561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064798799999999995,"stat_sum":{"num_bytes":32925991,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7147,"num_read_kb":50536,"num_write":1023,"num_write_kb":38907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.15","version":"231'613","reported_seq":6823,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606909+0000","last_change":"2024-04-24T00:37:46.604886+0000","last_active":"2024-04-24T00:37:53.606909+0000","last_peered":"2024-04-24T00:37:53.606909+0000","last_clean":"2024-04-24T00:37:53.606909+0000","last_became_active":"2024-04-24T00:35:28.725126+0000","last_became_peered":"2024-04-24T00:35:28.725126+0000","last_unstale":"2024-04-24T00:37:53.606909+0000","last_undegraded":"2024-04-24T00:37:53.606909+0000","last_fullsized":"2024-04-24T00:37:53.606909+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:18:28.502951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056773200000000005,"stat_sum":{"num_bytes":36441735,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":878,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":2312722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.14","version":"237'477","reported_seq":4997,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607116+0000","last_change":"2024-04-24T00:37:46.610711+0000","last_active":"2024-04-24T00:37:53.607116+0000","last_peered":"2024-04-24T00:37:53.607116+0000","last_clean":"2024-04-24T00:37:53.607116+0000","last_became_active":"2024-04-24T00:35:28.722992+0000","last_became_peered":"2024-04-24T00:35:28.722992+0000","last_unstale":"2024-04-24T00:37:53.607116+0000","last_undegraded":"2024-04-24T00:37:53.607116+0000","last_fullsized":"2024-04-24T00:37:53.607116+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:45:17.421114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058891000000000004,"stat_sum":{"num_bytes":19765972,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4140,"num_read_kb":41580,"num_write":566,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1929866,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.641374+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:37:53.641374+0000","last_peered":"2024-04-24T00:37:53.641374+0000","last_clean":"2024-04-24T00:37:53.641374+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:37:53.641374+0000","last_undegraded":"2024-04-24T00:37:53.641374+0000","last_fullsized":"2024-04-24T00:37:53.641374+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"232'529","reported_seq":6038,"reported_epoch":237,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-04-24T00:37:49.599990+0000","last_change":"2024-04-24T00:37:49.631656+0000","last_active":"2024-04-24T00:37:47.612198+0000","last_peered":"2024-04-24T00:37:46.617891+0000","last_clean":"2024-04-24T00:37:46.617891+0000","last_became_active":"2024-04-24T00:37:46.611693+0000","last_became_peered":"2024-04-24T00:37:46.611693+0000","last_unstale":"2024-04-24T00:37:49.599990+0000","last_undegraded":"2024-04-24T00:37:49.599990+0000","last_fullsized":"2024-04-24T00:37:49.595246+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001201978,"stat_sum":{"num_bytes":48215459,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5114,"num_read_kb":58057,"num_write":872,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":44596023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":24}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606313+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:37:53.606313+0000","last_peered":"2024-04-24T00:37:53.606313+0000","last_clean":"2024-04-24T00:37:53.606313+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:37:53.606313+0000","last_undegraded":"2024-04-24T00:37:53.606313+0000","last_fullsized":"2024-04-24T00:37:53.606313+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"231'580","reported_seq":4521,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.595609+0000","last_change":"2024-04-24T00:37:47.621693+0000","last_active":"2024-04-24T00:37:49.595609+0000","last_peered":"2024-04-24T00:37:49.595609+0000","last_clean":"2024-04-24T00:37:49.595609+0000","last_became_active":"2024-04-24T00:37:47.617077+0000","last_became_peered":"2024-04-24T00:37:47.617077+0000","last_unstale":"2024-04-24T00:37:49.595609+0000","last_undegraded":"2024-04-24T00:37:49.595609+0000","last_fullsized":"2024-04-24T00:37:49.595609+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:08:32.489010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002332027,"stat_sum":{"num_bytes":43812937,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4338,"num_read_kb":41656,"num_write":809,"num_write_kb":26241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.595036+0000","last_change":"2024-04-24T00:37:42.650295+0000","last_active":"2024-04-24T00:37:49.595036+0000","last_peered":"2024-04-24T00:37:49.595036+0000","last_clean":"2024-04-24T00:37:49.595036+0000","last_became_active":"2024-04-24T00:37:42.649872+0000","last_became_peered":"2024-04-24T00:37:42.649872+0000","last_unstale":"2024-04-24T00:37:49.595036+0000","last_undegraded":"2024-04-24T00:37:49.595036+0000","last_fullsized":"2024-04-24T00:37:49.595036+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:39:37.552779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"240'503","reported_seq":3936,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606384+0000","last_change":"2024-04-24T00:37:52.612184+0000","last_active":"2024-04-24T00:37:53.606384+0000","last_peered":"2024-04-24T00:37:53.606384+0000","last_clean":"2024-04-24T00:37:53.606384+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:37:53.606384+0000","last_undegraded":"2024-04-24T00:37:53.606384+0000","last_fullsized":"2024-04-24T00:37:53.606384+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:28:51.731025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082295399999999998,"stat_sum":{"num_bytes":21949422,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3436,"num_read_kb":34409,"num_write":707,"num_write_kb":26463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606261+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:37:53.606261+0000","last_peered":"2024-04-24T00:37:53.606261+0000","last_clean":"2024-04-24T00:37:53.606261+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:37:53.606261+0000","last_undegraded":"2024-04-24T00:37:53.606261+0000","last_fullsized":"2024-04-24T00:37:53.606261+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"237'618","reported_seq":5967,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.609490+0000","last_change":"2024-04-24T00:37:46.615072+0000","last_active":"2024-04-24T00:37:53.609490+0000","last_peered":"2024-04-24T00:37:53.609490+0000","last_clean":"2024-04-24T00:37:53.609490+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:37:53.609490+0000","last_undegraded":"2024-04-24T00:37:53.609490+0000","last_fullsized":"2024-04-24T00:37:53.609490+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:45:18.835496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002440181,"stat_sum":{"num_bytes":31195836,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5966,"num_read_kb":55481,"num_write":796,"num_write_kb":40841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.3","version":"240'481","reported_seq":6225,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607477+0000","last_change":"2024-04-24T00:37:47.629526+0000","last_active":"2024-04-24T00:37:53.607477+0000","last_peered":"2024-04-24T00:37:53.607477+0000","last_clean":"2024-04-24T00:37:53.607477+0000","last_became_active":"2024-04-24T00:37:47.623000+0000","last_became_peered":"2024-04-24T00:37:47.623000+0000","last_unstale":"2024-04-24T00:37:53.607477+0000","last_undegraded":"2024-04-24T00:37:53.607477+0000","last_fullsized":"2024-04-24T00:37:53.607477+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:41:21.537719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026935660000000001,"stat_sum":{"num_bytes":3134459,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4941,"num_read_kb":51362,"num_write":643,"num_write_kb":30821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":29414668,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":385,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608321+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:37:53.608321+0000","last_peered":"2024-04-24T00:37:53.608321+0000","last_clean":"2024-04-24T00:37:53.608321+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:37:53.608321+0000","last_undegraded":"2024-04-24T00:37:53.608321+0000","last_fullsized":"2024-04-24T00:37:53.608321+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"231'595","reported_seq":7569,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.616544+0000","last_change":"2024-04-24T00:37:53.613733+0000","last_active":"2024-04-24T00:37:53.616544+0000","last_peered":"2024-04-24T00:37:53.616544+0000","last_clean":"2024-04-24T00:37:53.616544+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:37:53.616544+0000","last_undegraded":"2024-04-24T00:37:53.616544+0000","last_fullsized":"2024-04-24T00:37:53.616544+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:58.963569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064418099999999996,"stat_sum":{"num_bytes":15384600,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7270,"num_read_kb":65726,"num_write":672,"num_write_kb":34854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":455,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608293+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:37:53.608293+0000","last_peered":"2024-04-24T00:37:53.608293+0000","last_clean":"2024-04-24T00:37:53.608293+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:37:53.608293+0000","last_undegraded":"2024-04-24T00:37:53.608293+0000","last_fullsized":"2024-04-24T00:37:53.608293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"231'426","reported_seq":4103,"reported_epoch":236,"state":"peering","last_fresh":"2024-04-24T00:37:48.610467+0000","last_change":"2024-04-24T00:37:49.596425+0000","last_active":"2024-04-24T00:37:49.596425+0000","last_peered":"2024-04-24T00:37:48.610467+0000","last_clean":"2024-04-24T00:37:48.610467+0000","last_became_active":"2024-04-24T00:37:45.607305+0000","last_became_peered":"2024-04-24T00:37:45.607305+0000","last_unstale":"2024-04-24T00:37:48.610467+0000","last_undegraded":"2024-04-24T00:37:48.610467+0000","last_fullsized":"2024-04-24T00:37:48.610467+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033054920000000002,"stat_sum":{"num_bytes":26338783,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2119,"num_read_kb":33775,"num_write":541,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":27566656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606074+0000","last_change":"2024-04-24T00:37:48.620512+0000","last_active":"2024-04-24T00:37:53.606074+0000","last_peered":"2024-04-24T00:37:53.606074+0000","last_clean":"2024-04-24T00:37:53.606074+0000","last_became_active":"2024-04-24T00:37:48.620159+0000","last_became_peered":"2024-04-24T00:37:48.620159+0000","last_unstale":"2024-04-24T00:37:53.606074+0000","last_undegraded":"2024-04-24T00:37:53.606074+0000","last_fullsized":"2024-04-24T00:37:53.606074+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:47.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"237'559","reported_seq":2574,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608643+0000","last_change":"2024-04-24T00:37:46.613641+0000","last_active":"2024-04-24T00:37:53.608643+0000","last_peered":"2024-04-24T00:37:53.608643+0000","last_clean":"2024-04-24T00:37:53.608643+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:37:53.608643+0000","last_undegraded":"2024-04-24T00:37:53.608643+0000","last_fullsized":"2024-04-24T00:37:53.608643+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:10.421466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019991789999999998,"stat_sum":{"num_bytes":22828543,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2553,"num_read_kb":24145,"num_write":733,"num_write_kb":27938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":385,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608544+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:37:53.608544+0000","last_peered":"2024-04-24T00:37:53.608544+0000","last_clean":"2024-04-24T00:37:53.608544+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:37:53.608544+0000","last_undegraded":"2024-04-24T00:37:53.608544+0000","last_fullsized":"2024-04-24T00:37:53.608544+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":405,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594456+0000","last_change":"2024-04-24T00:37:45.610754+0000","last_active":"2024-04-24T00:37:49.594456+0000","last_peered":"2024-04-24T00:37:49.594456+0000","last_clean":"2024-04-24T00:37:49.594456+0000","last_became_active":"2024-04-24T00:37:45.609794+0000","last_became_peered":"2024-04-24T00:37:45.609794+0000","last_unstale":"2024-04-24T00:37:49.594456+0000","last_undegraded":"2024-04-24T00:37:49.594456+0000","last_fullsized":"2024-04-24T00:37:49.594456+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:27.857798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"241'864","reported_seq":6640,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.627399+0000","last_change":"2024-04-24T00:37:53.614285+0000","last_active":"2024-04-24T00:37:53.627399+0000","last_peered":"2024-04-24T00:37:53.627399+0000","last_clean":"2024-04-24T00:37:53.627399+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:37:53.627399+0000","last_undegraded":"2024-04-24T00:37:53.627399+0000","last_fullsized":"2024-04-24T00:37:53.627399+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:44.142298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031730159999999999,"stat_sum":{"num_bytes":28042913,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7213,"num_read_kb":68776,"num_write":1184,"num_write_kb":32529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.9","version":"241'938","reported_seq":4829,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.624815+0000","last_change":"2024-04-24T00:37:46.607485+0000","last_active":"2024-04-24T00:37:53.624815+0000","last_peered":"2024-04-24T00:37:53.624815+0000","last_clean":"2024-04-24T00:37:53.624815+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:37:53.624815+0000","last_undegraded":"2024-04-24T00:37:53.624815+0000","last_fullsized":"2024-04-24T00:37:53.624815+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:55.142994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032049069999999999,"stat_sum":{"num_bytes":41212442,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":7165,"num_read_kb":32630,"num_write":1201,"num_write_kb":21153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.a","version":"232'1867","reported_seq":4124,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.640963+0000","last_change":"2024-04-24T00:37:46.609043+0000","last_active":"2024-04-24T00:37:53.640963+0000","last_peered":"2024-04-24T00:37:53.640963+0000","last_clean":"2024-04-24T00:37:53.640963+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:37:53.640963+0000","last_undegraded":"2024-04-24T00:37:53.640963+0000","last_fullsized":"2024-04-24T00:37:53.640963+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1867,"log_dups_size":0,"ondisk_log_size":1867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:47:20.613604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003056736,"stat_sum":{"num_bytes":22016457,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":9506,"num_read_kb":44890,"num_write":3222,"num_write_kb":33615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.b","version":"225'445","reported_seq":4121,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-24T00:37:49.594685+0000","last_change":"2024-04-24T00:37:46.603997+0000","last_active":"2024-04-24T00:37:49.594685+0000","last_peered":"2024-04-24T00:37:49.594685+0000","last_clean":"2024-04-24T00:37:49.594685+0000","last_became_active":"2024-04-24T00:37:45.607329+0000","last_became_peered":"2024-04-24T00:37:45.607329+0000","last_unstale":"2024-04-24T00:37:49.594685+0000","last_undegraded":"2024-04-24T00:37:49.594685+0000","last_fullsized":"2024-04-24T00:37:49.594685+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:18:59.283239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031718229999999998,"stat_sum":{"num_bytes":11945415,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3983,"num_read_kb":34787,"num_write":459,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.c","version":"237'755","reported_seq":9761,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608778+0000","last_change":"2024-04-24T00:37:46.614718+0000","last_active":"2024-04-24T00:37:53.608778+0000","last_peered":"2024-04-24T00:37:53.608778+0000","last_clean":"2024-04-24T00:37:53.608778+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:37:53.608778+0000","last_undegraded":"2024-04-24T00:37:53.608778+0000","last_fullsized":"2024-04-24T00:37:53.608778+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:39.565151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025462010000000001,"stat_sum":{"num_bytes":62470277,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10330,"num_read_kb":68570,"num_write":1104,"num_write_kb":35198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.d","version":"237'417","reported_seq":2752,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608451+0000","last_change":"2024-04-24T00:37:46.613660+0000","last_active":"2024-04-24T00:37:53.608451+0000","last_peered":"2024-04-24T00:37:53.608451+0000","last_clean":"2024-04-24T00:37:53.608451+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:37:53.608451+0000","last_undegraded":"2024-04-24T00:37:53.608451+0000","last_fullsized":"2024-04-24T00:37:53.608451+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:36.391472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015713649999999999,"stat_sum":{"num_bytes":3762622,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":547,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.e","version":"237'657","reported_seq":7668,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607299+0000","last_change":"2024-04-24T00:37:46.610598+0000","last_active":"2024-04-24T00:37:53.607299+0000","last_peered":"2024-04-24T00:37:53.607299+0000","last_clean":"2024-04-24T00:37:53.607299+0000","last_became_active":"2024-04-24T00:35:29.560215+0000","last_became_peered":"2024-04-24T00:35:29.560215+0000","last_unstale":"2024-04-24T00:37:53.607299+0000","last_undegraded":"2024-04-24T00:37:53.607299+0000","last_fullsized":"2024-04-24T00:37:53.607299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:53.583917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046475290000000001,"stat_sum":{"num_bytes":19901553,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":939,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":2728103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.f","version":"231'841","reported_seq":5456,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607426+0000","last_change":"2024-04-24T00:37:48.623765+0000","last_active":"2024-04-24T00:37:53.607426+0000","last_peered":"2024-04-24T00:37:53.607426+0000","last_clean":"2024-04-24T00:37:53.607426+0000","last_became_active":"2024-04-24T00:37:48.616654+0000","last_became_peered":"2024-04-24T00:37:48.616654+0000","last_unstale":"2024-04-24T00:37:53.607426+0000","last_undegraded":"2024-04-24T00:37:53.607426+0000","last_fullsized":"2024-04-24T00:37:53.607426+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:39:58.213721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037740320000000001,"stat_sum":{"num_bytes":27430936,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5510,"num_read_kb":46049,"num_write":1340,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":26495407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.10","version":"231'559","reported_seq":6105,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.641451+0000","last_change":"2024-04-24T00:37:46.615985+0000","last_active":"2024-04-24T00:37:53.641451+0000","last_peered":"2024-04-24T00:37:53.641451+0000","last_clean":"2024-04-24T00:37:53.641451+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:37:53.641451+0000","last_undegraded":"2024-04-24T00:37:53.641451+0000","last_fullsized":"2024-04-24T00:37:53.641451+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:24:18.385907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001356445,"stat_sum":{"num_bytes":10463442,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5981,"num_read_kb":30449,"num_write":663,"num_write_kb":20575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.11","version":"237'541","reported_seq":7940,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607904+0000","last_change":"2024-04-24T00:37:46.609789+0000","last_active":"2024-04-24T00:37:53.607904+0000","last_peered":"2024-04-24T00:37:53.607904+0000","last_clean":"2024-04-24T00:37:53.607904+0000","last_became_active":"2024-04-24T00:35:28.722568+0000","last_became_peered":"2024-04-24T00:35:28.722568+0000","last_unstale":"2024-04-24T00:37:53.607904+0000","last_undegraded":"2024-04-24T00:37:53.607904+0000","last_fullsized":"2024-04-24T00:37:53.607904+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:25.198938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053693430000000004,"stat_sum":{"num_bytes":29758977,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7209,"num_read_kb":84250,"num_write":643,"num_write_kb":47836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":1618865,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.12","version":"224'1450","reported_seq":5482,"reported_epoch":237,"state":"remapped+peering","last_fresh":"2024-04-24T00:37:49.595079+0000","last_change":"2024-04-24T00:37:49.595079+0000","last_active":"2024-04-24T00:37:49.595079+0000","last_peered":"2024-04-24T00:37:47.396304+0000","last_clean":"2024-04-24T00:37:40.621256+0000","last_became_active":"2024-04-24T00:37:43.717521+0000","last_became_peered":"2024-04-24T00:37:43.717521+0000","last_unstale":"2024-04-24T00:37:49.595079+0000","last_undegraded":"2024-04-24T00:37:49.595079+0000","last_fullsized":"2024-04-24T00:37:49.595079+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1450,"log_dups_size":0,"ondisk_log_size":1450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":41640823,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7403,"num_read_kb":53792,"num_write":2236,"num_write_kb":43767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":46534159,"num_keys_recovered":76,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"32","length":"1"},{"start":"34","length":"1"},{"start":"36","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.13","version":"232'638","reported_seq":4693,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607266+0000","last_change":"2024-04-24T00:37:46.612592+0000","last_active":"2024-04-24T00:37:53.607266+0000","last_peered":"2024-04-24T00:37:53.607266+0000","last_clean":"2024-04-24T00:37:53.607266+0000","last_became_active":"2024-04-24T00:35:28.722941+0000","last_became_peered":"2024-04-24T00:35:28.722941+0000","last_unstale":"2024-04-24T00:37:53.607266+0000","last_undegraded":"2024-04-24T00:37:53.607266+0000","last_fullsized":"2024-04-24T00:37:53.607266+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:28:06.433639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060203920000000003,"stat_sum":{"num_bytes":22578363,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4875,"num_read_kb":39170,"num_write":915,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":962033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":862168617,"num_objects":517,"num_object_clones":0,"num_object_copies":1034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":181225,"num_read_kb":1541757,"num_write":30009,"num_write_kb":1017899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5074,"num_bytes_recovered":326734394,"num_keys_recovered":482,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26759168,"data_stored":175790878,"data_compressed":9259835,"data_compressed_allocated":20037632,"data_compressed_original":172309093,"omap_allocated":4834,"internal_metadata":0},"log_size":21470,"ondisk_log_size":21470,"up":64,"acting":62,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":34064,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574904,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80764,"kb_used_data":5160,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291076,"statfs":{"total":96636764160,"available":96554061824,"internally_reserved":0,"allocated":5283840,"data_stored":16849329,"data_compressed":696513,"data_compressed_allocated":1662976,"data_compressed_original":15042905,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74292,"kb_used_data":6992,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297548,"statfs":{"total":96636764160,"available":96560689152,"internally_reserved":0,"allocated":7159808,"data_stored":29150018,"data_compressed":1492846,"data_compressed_allocated":3203072,"data_compressed_original":27176960,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82724,"kb_used_data":7132,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289116,"statfs":{"total":96636764160,"available":96552054784,"internally_reserved":0,"allocated":7303168,"data_stored":31027063,"data_compressed":1372796,"data_compressed_allocated":3211264,"data_compressed_original":28962528,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75040,"kb_used_data":7756,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296800,"statfs":{"total":96636764160,"available":96559923200,"internally_reserved":0,"allocated":7942144,"data_stored":32878848,"data_compressed":1923692,"data_compressed_allocated":3829760,"data_compressed_original":30814570,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57796,"kb_used_data":4408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314044,"statfs":{"total":96636764160,"available":96577581056,"internally_reserved":0,"allocated":4513792,"data_stored":11541922,"data_compressed":567944,"data_compressed_allocated":1175552,"data_compressed_original":9871360,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58184,"kb_used_data":4792,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94313656,"statfs":{"total":96636764160,"available":96577183744,"internally_reserved":0,"allocated":4907008,"data_stored":15667658,"data_compressed":596542,"data_compressed_allocated":1466368,"data_compressed_original":13991936,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893152,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35332,"kb_used_data":7600,"kb_used_omap":6,"kb_used_meta":27705,"kb_avail":94336508,"statfs":{"total":96636764160,"available":96600584192,"internally_reserved":0,"allocated":7782400,"data_stored":37064126,"data_compressed":1713150,"data_compressed_allocated":3862528,"data_compressed_original":35086336,"omap_allocated":6429,"internal_metadata":28370659},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4882432,"data_stored":35057512,"data_compressed":1706165,"data_compressed_allocated":3825664,"data_compressed_original":34496512,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1998848,"data_stored":14226714,"data_compressed":596542,"data_compressed_allocated":1466368,"data_compressed_original":13991936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1605632,"data_stored":10100978,"data_compressed":567944,"data_compressed_allocated":1175552,"data_compressed_original":9871360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5033984,"data_stored":31437904,"data_compressed":1923692,"data_compressed_allocated":3829760,"data_compressed_original":30814570,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4059136,"data_stored":27024800,"data_compressed":1282788,"data_compressed_allocated":2961408,"data_compressed_original":26439392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3313664,"data_stored":19507273,"data_compressed":1220787,"data_compressed_allocated":2433024,"data_compressed_original":19066880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2166784,"data_stored":13485231,"data_compressed":675403,"data_compressed_allocated":1527808,"data_compressed_original":13146457,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:55.177 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:55.174+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.255 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.54389267600001, 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:37:55.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:37:55.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:37:55.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:37:55.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:55.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:37:55.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:37:55.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:37:55.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:37:55.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:55.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:37:55.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:37:55.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.264 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.265 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.265 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:55.265 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:55.265 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.54379475100001, 2024-04-24T00:37:55.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:37:55.265 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.266 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.266 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.266 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.266 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.266 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.266 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:37:55.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.266 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:55.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:37:55.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:37:55.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:37:55.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:37:55.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:37:55.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:37:55.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.274 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 194.18441521899999, 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:37:55.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.280 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 170.77367109299999, 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:37:55.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:37:55.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:37:55.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:37:55.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 170.74840248699999, 2024-04-24T00:37:55.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:37:55.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.89519876700001, 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:37:55.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:37:55.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:37:55.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:37:55.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:37:55.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:37:55.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:37:55.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:37:55.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:37:55.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:37:55.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:37:55.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:37:55.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:55.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:37:55.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:37:55.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:37:55.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:37:55.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:37:55.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:37:55.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:37:55.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:37:55.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:37:55.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:37:55.301 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.301 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.301 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.301 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:37:55.301 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:55.302 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.862593653, 2024-04-24T00:37:55.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:37:55.302 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.302 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.302 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.302 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:55.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:37:55.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:37:55.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:37:55.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:37:55.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:37:55.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:37:55.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:37:55.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:37:55.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:37:55.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:37:55.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:37:55.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:37:55.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:37:55.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:37:55.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.310 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.310 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:55.310 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:55.310 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.45801895899999, 2024-04-24T00:37:55.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:37:55.311 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.311 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.311 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.311 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.311 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.312 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:37:55.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.312 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:55.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:55.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:37:55.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:37:55.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:37:55.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:37:55.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:37:55.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:37:55.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:37:55.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:37:55.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:37:55.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:37:55.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:37:55.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:37:55.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:55.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:37:55.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:37:55.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:37:55.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:55.322 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 156.400178729, 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.326 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.57986882900001, 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:55.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:37:55.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:37:55.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.332 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.550226776, 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:55.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.336 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.40371136500001, 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.337 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.614938807, 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.343 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:37:55.348 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.591795394, 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:37:55.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:37:55.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.718486646000002, 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.353 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:55.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:37:55.358 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.665804369, 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:37:55.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:166979 3.5 3:a3485662:::rbd_data.115e4ca9eff1.0000000000000010:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.266072416, 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.145524811, 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 166979 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986190+0000", 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.323e-06 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986212+0000", 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1183999999999999e-05 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986246+0000", 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4409999999999998e-05 2024-04-24T00:37:55.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986265+0000", 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986437+0000", 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017130199999999999 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986581+0000", 2024-04-24T00:37:55.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014409500000000001 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.997390+0000", 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010809456 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131432+0000", 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.134041997 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131521+0000", 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9024999999999996e-05 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131625+0000", 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103923 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131713+0000", 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7672999999999999e-05 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:166979 3.5 e206/56 ondisk, result = 0)", 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.222537447999997, 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10200487, 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029726+0000", 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4899999999999999e-06 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029738+0000", 2024-04-24T00:37:55.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2483000000000001e-05 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029784+0000", 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5920000000000001e-05 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029809+0000", 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5193000000000001e-05 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131491+0000", 2024-04-24T00:37:55.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101681585 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131510+0000", 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9256999999999999e-05 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131728+0000", 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021794199999999999 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169782 3.d 3:b3c6180a:::rbd_data.115ee24d4bb.000000000000000f:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.119414908000003, 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.117826449, 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169782 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.373 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132848+0000", 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0770000000000002e-06 2024-04-24T00:37:55.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132859+0000", 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132887+0000", 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8070000000000001e-05 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132908+0000", 2024-04-24T00:37:55.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.09e-05 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133079+0000", 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017107900000000001 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133247+0000", 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168138 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.141250+0000", 2024-04-24T00:37:55.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080027740000000007 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250223+0000", 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108972528 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250437+0000", 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021457900000000001 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:55.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250540+0000", 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102682 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250672+0000", 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013205899999999999 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169782 3.d e210/106 ondisk, result = 0)", 2024-04-24T00:37:55.378 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.090498576000002, 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088930206999999997, 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:55.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161764+0000", 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3259999999999998e-06 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161773+0000", 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5329999999999994e-06 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:55.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161803+0000", 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9867000000000001e-05 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161825+0000", 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2082999999999998e-05 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250323+0000", 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088498544999999998 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:55.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250342+0000", 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8983000000000001e-05 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250692+0000", 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034987000000000001 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:55.382 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:55.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:55.384 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:55.382+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:55.485 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:55.482+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:55.511 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:55.512 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:55.512 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:55.512 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:55.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:37:55.586 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:55.582+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:55.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:37:55.687 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:55.682+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:55.695 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:55.696 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:55.696 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:55.696 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:55.696 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:55.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:37:55.755 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:55.755 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:55.772 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:55.772 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:55.791 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:55.791 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:55.796 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:55.794+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:55.807 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:55.808 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:55.830 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:55.830 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:55.851 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:55.851 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:55.871 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:55.872 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:55.881 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:55.881 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:55.889 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:55.886+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:55.892 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:55.892 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:55.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:37:55.989 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:55.986+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:56.065 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:56.065 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:56.079 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:37:56.120 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:56.118+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:56.192 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:56.186+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:56.241 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:56.241 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:56.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:37:56.293 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:56.290+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:56.394 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:56.390+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:56.421 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:56.421 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:56.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:37:56.495 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:56.490+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:56.570 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:56.570 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:56.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:37:56.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:37:56.633 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:56.630+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:56.697 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:56.694+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:56.712 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:56.712 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:56.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:37:56.798 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:56.794+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:56.899 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:56.894+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:56.910 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:56.910 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:56.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:37:57.013 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:57.010+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:57.085 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:57.085 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:57.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:37:57.101 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:57.098+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:57.201 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:57.198+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:57.265 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:57.266 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:57.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:37:57.321 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:57.318+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:57.404 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:57.398+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:57.449 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:57.449 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:57.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:37:57.505 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:57.502+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:57.606 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:57.602+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:57.628 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:57.628 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:57.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:37:57.729 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:57.726+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:57.740 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:37:57.813 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:57.810+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:57.821 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:57.821 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:57.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:37:57.910 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:57.906+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:57.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:37:58.011 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:58.006+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:37:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:37:58.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:37:58.071 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:37:58.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:37:58.089 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:37:58.089 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:37:58.106 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:37:58.106 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:37:58.112 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:58.106+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:58.123 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:37:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:37:58.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:37:58.144 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:37:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:37:58.165 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:37:58.165 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:37:58.186 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:37:58.186 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:58.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:37:58.208 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:37:58.209 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:37:58.213 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:58.210+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:58.314 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:37:58.310+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:37:58.326 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:37:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:37:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:37:58.327 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:58.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:37:58.434 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:58.430+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:58.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-24T00:37:58.458+0000 7f0246143640 0 --2- 172.21.15.64:0/4096880065 >> [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] conn(0x7f02401402e0 0x7f02401d74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-24T00:37:58.514 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:37:58.514 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.29961766, 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.515 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:37:58.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.29926674500001, 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:37:58.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:37:58.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.29237842500001, 2024-04-24T00:37:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:37:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:37:58.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:58.530 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 174.034096799, 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:58.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:37:58.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:37:58.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:37:58.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 174.02382038299999, 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:37:58.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:37:58.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:37:58.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.56186958000001, 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:37:58.541 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:37:58.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.53583555899999, 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:37:58.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:37:58.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.321892652, 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:58.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:37:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.31188070799999, 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:37:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:37:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:37:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.038501846, 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:37:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.02183339699999, 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:37:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.83593561399999, 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:37:58.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.15217725700001, 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 116.129960259, 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 102.82539042099999, 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 102.812765461, 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:37:58.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.463273974000003, 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:37:58.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:37:58.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.448255779, 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:37:58.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.555439376000002, 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:37:58.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:37:58.613 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.518087348999998, 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:37:58.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:37:58.617 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:37:58.618 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:37:58.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:37:58.620 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:58.614+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:58.648 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:37:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:37:58.667 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-24T00:37:57.735975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":837724191,"num_objects":508,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":181343,"num_read_kb":1541858,"num_write":30179,"num_write_kb":1020315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5448,"num_bytes_recovered":338832835,"num_keys_recovered":493,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21599,"ondisk_log_size":21599,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":463560,"kb_used_data":43272,"kb_used_omap":15,"kb_used_meta":420144,"kb_avail":660139320,"statfs":{"total":676457349120,"available":675982663680,"internally_reserved":0,"allocated":44310528,"data_stored":165181149,"data_compressed":8305140,"data_compressed_allocated":17801216,"data_compressed_original":151935395,"omap_allocated":15996,"internal_metadata":430227844},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-65396126,"num_objects":-31,"num_object_clones":0,"num_object_copies":-62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-31,"num_whiteouts":0,"num_read":134,"num_read_kb":111,"num_write":48,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":18104684,"num_keys_recovered":54,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.022373"},"pg_stats":[{"pgid":"3.1f","version":"237'833","reported_seq":5691,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.600002+0000","last_change":"2024-04-24T00:37:54.600664+0000","last_active":"2024-04-24T00:37:54.600002+0000","last_peered":"2024-04-24T00:37:54.600002+0000","last_clean":"2024-04-24T00:37:54.600002+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:37:54.600002+0000","last_undegraded":"2024-04-24T00:37:54.600002+0000","last_fullsized":"2024-04-24T00:37:54.600002+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":833,"log_dups_size":0,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:47:27.658552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034759069999999999,"stat_sum":{"num_bytes":45731760,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":6142,"num_read_kb":55851,"num_write":1310,"num_write_kb":38951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"}]},{"pgid":"3.1e","version":"231'977","reported_seq":5301,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.641642+0000","last_change":"2024-04-24T00:37:46.609936+0000","last_active":"2024-04-24T00:37:53.641642+0000","last_peered":"2024-04-24T00:37:53.641642+0000","last_clean":"2024-04-24T00:37:53.641642+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:37:53.641642+0000","last_undegraded":"2024-04-24T00:37:53.641642+0000","last_fullsized":"2024-04-24T00:37:53.641642+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:42.516498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003187456,"stat_sum":{"num_bytes":21004221,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7576,"num_read_kb":34899,"num_write":1269,"num_write_kb":26860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1d","version":"232'552","reported_seq":6745,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.609139+0000","last_change":"2024-04-24T00:37:46.614489+0000","last_active":"2024-04-24T00:37:53.609139+0000","last_peered":"2024-04-24T00:37:53.609139+0000","last_clean":"2024-04-24T00:37:53.609139+0000","last_became_active":"2024-04-24T00:34:28.726196+0000","last_became_peered":"2024-04-24T00:34:28.726196+0000","last_unstale":"2024-04-24T00:37:53.609139+0000","last_undegraded":"2024-04-24T00:37:53.609139+0000","last_fullsized":"2024-04-24T00:37:53.609139+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:00:13.573517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001963238,"stat_sum":{"num_bytes":32555141,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7132,"num_read_kb":66702,"num_write":793,"num_write_kb":27396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":458933,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1c","version":"240'718","reported_seq":5410,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608467+0000","last_change":"2024-04-24T00:37:46.613357+0000","last_active":"2024-04-24T00:37:53.608467+0000","last_peered":"2024-04-24T00:37:53.608467+0000","last_clean":"2024-04-24T00:37:53.608467+0000","last_became_active":"2024-04-24T00:34:31.747052+0000","last_became_peered":"2024-04-24T00:34:31.747052+0000","last_unstale":"2024-04-24T00:37:53.608467+0000","last_undegraded":"2024-04-24T00:37:53.608467+0000","last_fullsized":"2024-04-24T00:37:53.608467+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:11.238051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001537114,"stat_sum":{"num_bytes":31150299,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":4957,"num_read_kb":61091,"num_write":917,"num_write_kb":35125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":949407,"num_keys_recovered":27,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1b","version":"231'455","reported_seq":4244,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608451+0000","last_change":"2024-04-24T00:37:46.611866+0000","last_active":"2024-04-24T00:37:53.608451+0000","last_peered":"2024-04-24T00:37:53.608451+0000","last_clean":"2024-04-24T00:37:53.608451+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:37:53.608451+0000","last_undegraded":"2024-04-24T00:37:53.608451+0000","last_fullsized":"2024-04-24T00:37:53.608451+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:08.897008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056302700000000002,"stat_sum":{"num_bytes":26704680,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3990,"num_read_kb":45455,"num_write":567,"num_write_kb":27383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.1a","version":"232'374","reported_seq":5743,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606727+0000","last_change":"2024-04-24T00:37:51.619432+0000","last_active":"2024-04-24T00:37:53.606727+0000","last_peered":"2024-04-24T00:37:53.606727+0000","last_clean":"2024-04-24T00:37:53.606727+0000","last_became_active":"2024-04-24T00:37:51.618422+0000","last_became_peered":"2024-04-24T00:37:51.618422+0000","last_unstale":"2024-04-24T00:37:53.606727+0000","last_undegraded":"2024-04-24T00:37:53.606727+0000","last_fullsized":"2024-04-24T00:37:53.606727+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:29:27.578800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048149600000000001,"stat_sum":{"num_bytes":9349759,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4390,"num_read_kb":37215,"num_write":485,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":2899650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.19","version":"237'681","reported_seq":5676,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606240+0000","last_change":"2024-04-24T00:37:52.617102+0000","last_active":"2024-04-24T00:37:53.606240+0000","last_peered":"2024-04-24T00:37:53.606240+0000","last_clean":"2024-04-24T00:37:53.606240+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:37:53.606240+0000","last_undegraded":"2024-04-24T00:37:53.606240+0000","last_fullsized":"2024-04-24T00:37:53.606240+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:05:48.471540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000930561,"stat_sum":{"num_bytes":22788004,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5927,"num_read_kb":40381,"num_write":976,"num_write_kb":29815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.18","version":"237'538","reported_seq":4725,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.598691+0000","last_change":"2024-04-24T00:37:54.599105+0000","last_active":"2024-04-24T00:37:54.598691+0000","last_peered":"2024-04-24T00:37:54.598691+0000","last_clean":"2024-04-24T00:37:54.598691+0000","last_became_active":"2024-04-24T00:37:46.608136+0000","last_became_peered":"2024-04-24T00:37:46.608136+0000","last_unstale":"2024-04-24T00:37:54.598691+0000","last_undegraded":"2024-04-24T00:37:54.598691+0000","last_fullsized":"2024-04-24T00:37:54.598691+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:16.760046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025962670000000002,"stat_sum":{"num_bytes":24759841,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":585,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"}]},{"pgid":"3.17","version":"231'322","reported_seq":4608,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.601060+0000","last_change":"2024-04-24T00:37:54.601878+0000","last_active":"2024-04-24T00:37:54.601060+0000","last_peered":"2024-04-24T00:37:54.601060+0000","last_clean":"2024-04-24T00:37:54.601060+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:37:54.601060+0000","last_undegraded":"2024-04-24T00:37:54.601060+0000","last_fullsized":"2024-04-24T00:37:54.601060+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:12:30.597058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042410679999999997,"stat_sum":{"num_bytes":14945483,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":417,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"}]},{"pgid":"3.16","version":"231'670","reported_seq":6329,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607040+0000","last_change":"2024-04-24T00:37:46.605862+0000","last_active":"2024-04-24T00:37:53.607040+0000","last_peered":"2024-04-24T00:37:53.607040+0000","last_clean":"2024-04-24T00:37:53.607040+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:37:53.607040+0000","last_undegraded":"2024-04-24T00:37:53.607040+0000","last_fullsized":"2024-04-24T00:37:53.607040+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:28:04.964561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064798799999999995,"stat_sum":{"num_bytes":32925991,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7147,"num_read_kb":50536,"num_write":1023,"num_write_kb":38907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.15","version":"231'613","reported_seq":6823,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606909+0000","last_change":"2024-04-24T00:37:46.604886+0000","last_active":"2024-04-24T00:37:53.606909+0000","last_peered":"2024-04-24T00:37:53.606909+0000","last_clean":"2024-04-24T00:37:53.606909+0000","last_became_active":"2024-04-24T00:35:28.725126+0000","last_became_peered":"2024-04-24T00:35:28.725126+0000","last_unstale":"2024-04-24T00:37:53.606909+0000","last_undegraded":"2024-04-24T00:37:53.606909+0000","last_fullsized":"2024-04-24T00:37:53.606909+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:18:28.502951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056773200000000005,"stat_sum":{"num_bytes":36441735,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":878,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":2312722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.14","version":"237'477","reported_seq":4997,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607116+0000","last_change":"2024-04-24T00:37:46.610711+0000","last_active":"2024-04-24T00:37:53.607116+0000","last_peered":"2024-04-24T00:37:53.607116+0000","last_clean":"2024-04-24T00:37:53.607116+0000","last_became_active":"2024-04-24T00:35:28.722992+0000","last_became_peered":"2024-04-24T00:35:28.722992+0000","last_unstale":"2024-04-24T00:37:53.607116+0000","last_undegraded":"2024-04-24T00:37:53.607116+0000","last_fullsized":"2024-04-24T00:37:53.607116+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:45:17.421114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058891000000000004,"stat_sum":{"num_bytes":19765972,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4140,"num_read_kb":41580,"num_write":566,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":1929866,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.641374+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:37:53.641374+0000","last_peered":"2024-04-24T00:37:53.641374+0000","last_clean":"2024-04-24T00:37:53.641374+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:37:53.641374+0000","last_undegraded":"2024-04-24T00:37:53.641374+0000","last_fullsized":"2024-04-24T00:37:53.641374+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"240'533","reported_seq":6286,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.599255+0000","last_change":"2024-04-24T00:37:54.599540+0000","last_active":"2024-04-24T00:37:54.599255+0000","last_peered":"2024-04-24T00:37:54.599255+0000","last_clean":"2024-04-24T00:37:54.599255+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:37:54.599255+0000","last_undegraded":"2024-04-24T00:37:54.599255+0000","last_fullsized":"2024-04-24T00:37:54.599255+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:39.973773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029119800000000002,"stat_sum":{"num_bytes":41563555,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5114,"num_read_kb":58057,"num_write":875,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606313+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:37:53.606313+0000","last_peered":"2024-04-24T00:37:53.606313+0000","last_clean":"2024-04-24T00:37:53.606313+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:37:53.606313+0000","last_undegraded":"2024-04-24T00:37:53.606313+0000","last_fullsized":"2024-04-24T00:37:53.606313+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"231'580","reported_seq":4536,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.600739+0000","last_change":"2024-04-24T00:37:54.600958+0000","last_active":"2024-04-24T00:37:54.600739+0000","last_peered":"2024-04-24T00:37:54.600739+0000","last_clean":"2024-04-24T00:37:54.600739+0000","last_became_active":"2024-04-24T00:37:47.617077+0000","last_became_peered":"2024-04-24T00:37:47.617077+0000","last_unstale":"2024-04-24T00:37:54.600739+0000","last_undegraded":"2024-04-24T00:37:54.600739+0000","last_fullsized":"2024-04-24T00:37:54.600739+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:00:29.725106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034174190000000001,"stat_sum":{"num_bytes":43812937,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4338,"num_read_kb":41656,"num_write":809,"num_write_kb":26241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.596323+0000","last_change":"2024-04-24T00:37:42.650295+0000","last_active":"2024-04-24T00:37:54.596323+0000","last_peered":"2024-04-24T00:37:54.596323+0000","last_clean":"2024-04-24T00:37:54.596323+0000","last_became_active":"2024-04-24T00:37:42.649872+0000","last_became_peered":"2024-04-24T00:37:42.649872+0000","last_unstale":"2024-04-24T00:37:54.596323+0000","last_undegraded":"2024-04-24T00:37:54.596323+0000","last_fullsized":"2024-04-24T00:37:54.596323+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:39:37.552779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"240'503","reported_seq":3936,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606384+0000","last_change":"2024-04-24T00:37:52.612184+0000","last_active":"2024-04-24T00:37:53.606384+0000","last_peered":"2024-04-24T00:37:53.606384+0000","last_clean":"2024-04-24T00:37:53.606384+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:37:53.606384+0000","last_undegraded":"2024-04-24T00:37:53.606384+0000","last_fullsized":"2024-04-24T00:37:53.606384+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:28:51.731025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082295399999999998,"stat_sum":{"num_bytes":21949422,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3436,"num_read_kb":34409,"num_write":707,"num_write_kb":26463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606261+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:37:53.606261+0000","last_peered":"2024-04-24T00:37:53.606261+0000","last_clean":"2024-04-24T00:37:53.606261+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:37:53.606261+0000","last_undegraded":"2024-04-24T00:37:53.606261+0000","last_fullsized":"2024-04-24T00:37:53.606261+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"237'618","reported_seq":5967,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.609490+0000","last_change":"2024-04-24T00:37:46.615072+0000","last_active":"2024-04-24T00:37:53.609490+0000","last_peered":"2024-04-24T00:37:53.609490+0000","last_clean":"2024-04-24T00:37:53.609490+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:37:53.609490+0000","last_undegraded":"2024-04-24T00:37:53.609490+0000","last_fullsized":"2024-04-24T00:37:53.609490+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:45:18.835496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002440181,"stat_sum":{"num_bytes":31195836,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5966,"num_read_kb":55481,"num_write":796,"num_write_kb":40841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.3","version":"240'481","reported_seq":6225,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607477+0000","last_change":"2024-04-24T00:37:47.629526+0000","last_active":"2024-04-24T00:37:53.607477+0000","last_peered":"2024-04-24T00:37:53.607477+0000","last_clean":"2024-04-24T00:37:53.607477+0000","last_became_active":"2024-04-24T00:37:47.623000+0000","last_became_peered":"2024-04-24T00:37:47.623000+0000","last_unstale":"2024-04-24T00:37:53.607477+0000","last_undegraded":"2024-04-24T00:37:53.607477+0000","last_fullsized":"2024-04-24T00:37:53.607477+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:41:21.537719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026935660000000001,"stat_sum":{"num_bytes":3134459,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4941,"num_read_kb":51362,"num_write":643,"num_write_kb":30821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":29414668,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":385,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608321+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:37:53.608321+0000","last_peered":"2024-04-24T00:37:53.608321+0000","last_clean":"2024-04-24T00:37:53.608321+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:37:53.608321+0000","last_undegraded":"2024-04-24T00:37:53.608321+0000","last_fullsized":"2024-04-24T00:37:53.608321+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"231'595","reported_seq":7569,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.616544+0000","last_change":"2024-04-24T00:37:53.613733+0000","last_active":"2024-04-24T00:37:53.616544+0000","last_peered":"2024-04-24T00:37:53.616544+0000","last_clean":"2024-04-24T00:37:53.616544+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:37:53.616544+0000","last_undegraded":"2024-04-24T00:37:53.616544+0000","last_fullsized":"2024-04-24T00:37:53.616544+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:58.963569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064418099999999996,"stat_sum":{"num_bytes":15384600,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7270,"num_read_kb":65726,"num_write":672,"num_write_kb":34854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":455,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608293+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:37:53.608293+0000","last_peered":"2024-04-24T00:37:53.608293+0000","last_clean":"2024-04-24T00:37:53.608293+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:37:53.608293+0000","last_undegraded":"2024-04-24T00:37:53.608293+0000","last_fullsized":"2024-04-24T00:37:53.608293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"241'429","reported_seq":4360,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.600412+0000","last_change":"2024-04-24T00:37:54.601041+0000","last_active":"2024-04-24T00:37:54.600412+0000","last_peered":"2024-04-24T00:37:54.600412+0000","last_clean":"2024-04-24T00:37:54.600412+0000","last_became_active":"2024-04-24T00:37:53.608821+0000","last_became_peered":"2024-04-24T00:37:53.608821+0000","last_unstale":"2024-04-24T00:37:54.600412+0000","last_undegraded":"2024-04-24T00:37:54.600412+0000","last_fullsized":"2024-04-24T00:37:54.600412+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:01:11.405409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035583469999999999,"stat_sum":{"num_bytes":22717877,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":544,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":526,"num_bytes_recovered":30192524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.606074+0000","last_change":"2024-04-24T00:37:48.620512+0000","last_active":"2024-04-24T00:37:53.606074+0000","last_peered":"2024-04-24T00:37:53.606074+0000","last_clean":"2024-04-24T00:37:53.606074+0000","last_became_active":"2024-04-24T00:37:48.620159+0000","last_became_peered":"2024-04-24T00:37:48.620159+0000","last_unstale":"2024-04-24T00:37:53.606074+0000","last_undegraded":"2024-04-24T00:37:53.606074+0000","last_fullsized":"2024-04-24T00:37:53.606074+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:47.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"237'559","reported_seq":2574,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608643+0000","last_change":"2024-04-24T00:37:46.613641+0000","last_active":"2024-04-24T00:37:53.608643+0000","last_peered":"2024-04-24T00:37:53.608643+0000","last_clean":"2024-04-24T00:37:53.608643+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:37:53.608643+0000","last_undegraded":"2024-04-24T00:37:53.608643+0000","last_fullsized":"2024-04-24T00:37:53.608643+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:10.421466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019991789999999998,"stat_sum":{"num_bytes":22828543,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2553,"num_read_kb":24145,"num_write":733,"num_write_kb":27938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":385,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608544+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:37:53.608544+0000","last_peered":"2024-04-24T00:37:53.608544+0000","last_clean":"2024-04-24T00:37:53.608544+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:37:53.608544+0000","last_undegraded":"2024-04-24T00:37:53.608544+0000","last_fullsized":"2024-04-24T00:37:53.608544+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":415,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.596049+0000","last_change":"2024-04-24T00:37:45.610754+0000","last_active":"2024-04-24T00:37:54.596049+0000","last_peered":"2024-04-24T00:37:54.596049+0000","last_clean":"2024-04-24T00:37:54.596049+0000","last_became_active":"2024-04-24T00:37:45.609794+0000","last_became_peered":"2024-04-24T00:37:45.609794+0000","last_unstale":"2024-04-24T00:37:54.596049+0000","last_undegraded":"2024-04-24T00:37:54.596049+0000","last_fullsized":"2024-04-24T00:37:54.596049+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:27.857798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"241'864","reported_seq":6640,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.627399+0000","last_change":"2024-04-24T00:37:53.614285+0000","last_active":"2024-04-24T00:37:53.627399+0000","last_peered":"2024-04-24T00:37:53.627399+0000","last_clean":"2024-04-24T00:37:53.627399+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:37:53.627399+0000","last_undegraded":"2024-04-24T00:37:53.627399+0000","last_fullsized":"2024-04-24T00:37:53.627399+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:44.142298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031730159999999999,"stat_sum":{"num_bytes":28042913,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7213,"num_read_kb":68776,"num_write":1184,"num_write_kb":32529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.9","version":"241'938","reported_seq":4829,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.624815+0000","last_change":"2024-04-24T00:37:46.607485+0000","last_active":"2024-04-24T00:37:53.624815+0000","last_peered":"2024-04-24T00:37:53.624815+0000","last_clean":"2024-04-24T00:37:53.624815+0000","last_became_active":"2024-04-24T00:33:53.025336+0000","last_became_peered":"2024-04-24T00:33:53.025336+0000","last_unstale":"2024-04-24T00:37:53.624815+0000","last_undegraded":"2024-04-24T00:37:53.624815+0000","last_fullsized":"2024-04-24T00:37:53.624815+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:04:55.142994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032049069999999999,"stat_sum":{"num_bytes":41212442,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":7165,"num_read_kb":32630,"num_write":1201,"num_write_kb":21153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.a","version":"232'1867","reported_seq":4124,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.640963+0000","last_change":"2024-04-24T00:37:46.609043+0000","last_active":"2024-04-24T00:37:53.640963+0000","last_peered":"2024-04-24T00:37:53.640963+0000","last_clean":"2024-04-24T00:37:53.640963+0000","last_became_active":"2024-04-24T00:33:53.031924+0000","last_became_peered":"2024-04-24T00:33:53.031924+0000","last_unstale":"2024-04-24T00:37:53.640963+0000","last_undegraded":"2024-04-24T00:37:53.640963+0000","last_fullsized":"2024-04-24T00:37:53.640963+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1867,"log_dups_size":0,"ondisk_log_size":1867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:47:20.613604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003056736,"stat_sum":{"num_bytes":22016457,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":9506,"num_read_kb":44890,"num_write":3222,"num_write_kb":33615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.b","version":"225'445","reported_seq":4136,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.599765+0000","last_change":"2024-04-24T00:37:54.600181+0000","last_active":"2024-04-24T00:37:54.599765+0000","last_peered":"2024-04-24T00:37:54.599765+0000","last_clean":"2024-04-24T00:37:54.599765+0000","last_became_active":"2024-04-24T00:37:45.607329+0000","last_became_peered":"2024-04-24T00:37:45.607329+0000","last_unstale":"2024-04-24T00:37:54.599765+0000","last_undegraded":"2024-04-24T00:37:54.599765+0000","last_fullsized":"2024-04-24T00:37:54.599765+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:03.457630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00319156,"stat_sum":{"num_bytes":11945415,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3983,"num_read_kb":34787,"num_write":459,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"}]},{"pgid":"3.c","version":"237'755","reported_seq":9761,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608778+0000","last_change":"2024-04-24T00:37:46.614718+0000","last_active":"2024-04-24T00:37:53.608778+0000","last_peered":"2024-04-24T00:37:53.608778+0000","last_clean":"2024-04-24T00:37:53.608778+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:37:53.608778+0000","last_undegraded":"2024-04-24T00:37:53.608778+0000","last_fullsized":"2024-04-24T00:37:53.608778+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:39.565151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025462010000000001,"stat_sum":{"num_bytes":62470277,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10330,"num_read_kb":68570,"num_write":1104,"num_write_kb":35198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.d","version":"237'417","reported_seq":2752,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.608451+0000","last_change":"2024-04-24T00:37:46.613660+0000","last_active":"2024-04-24T00:37:53.608451+0000","last_peered":"2024-04-24T00:37:53.608451+0000","last_clean":"2024-04-24T00:37:53.608451+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:37:53.608451+0000","last_undegraded":"2024-04-24T00:37:53.608451+0000","last_fullsized":"2024-04-24T00:37:53.608451+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:36.391472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015713649999999999,"stat_sum":{"num_bytes":3762622,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":547,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.e","version":"237'657","reported_seq":7668,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607299+0000","last_change":"2024-04-24T00:37:46.610598+0000","last_active":"2024-04-24T00:37:53.607299+0000","last_peered":"2024-04-24T00:37:53.607299+0000","last_clean":"2024-04-24T00:37:53.607299+0000","last_became_active":"2024-04-24T00:35:29.560215+0000","last_became_peered":"2024-04-24T00:35:29.560215+0000","last_unstale":"2024-04-24T00:37:53.607299+0000","last_undegraded":"2024-04-24T00:37:53.607299+0000","last_fullsized":"2024-04-24T00:37:53.607299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:53.583917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046475290000000001,"stat_sum":{"num_bytes":19901553,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":939,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":2728103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.f","version":"231'841","reported_seq":5456,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607426+0000","last_change":"2024-04-24T00:37:48.623765+0000","last_active":"2024-04-24T00:37:53.607426+0000","last_peered":"2024-04-24T00:37:53.607426+0000","last_clean":"2024-04-24T00:37:53.607426+0000","last_became_active":"2024-04-24T00:37:48.616654+0000","last_became_peered":"2024-04-24T00:37:48.616654+0000","last_unstale":"2024-04-24T00:37:53.607426+0000","last_undegraded":"2024-04-24T00:37:53.607426+0000","last_fullsized":"2024-04-24T00:37:53.607426+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:39:58.213721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037740320000000001,"stat_sum":{"num_bytes":27430936,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5510,"num_read_kb":46049,"num_write":1340,"num_write_kb":43855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":26495407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.10","version":"231'559","reported_seq":6105,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.641451+0000","last_change":"2024-04-24T00:37:46.615985+0000","last_active":"2024-04-24T00:37:53.641451+0000","last_peered":"2024-04-24T00:37:53.641451+0000","last_clean":"2024-04-24T00:37:53.641451+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:37:53.641451+0000","last_undegraded":"2024-04-24T00:37:53.641451+0000","last_fullsized":"2024-04-24T00:37:53.641451+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:24:18.385907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001356445,"stat_sum":{"num_bytes":10463442,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5981,"num_read_kb":30449,"num_write":663,"num_write_kb":20575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.11","version":"237'541","reported_seq":7940,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607904+0000","last_change":"2024-04-24T00:37:46.609789+0000","last_active":"2024-04-24T00:37:53.607904+0000","last_peered":"2024-04-24T00:37:53.607904+0000","last_clean":"2024-04-24T00:37:53.607904+0000","last_became_active":"2024-04-24T00:35:28.722568+0000","last_became_peered":"2024-04-24T00:35:28.722568+0000","last_unstale":"2024-04-24T00:37:53.607904+0000","last_undegraded":"2024-04-24T00:37:53.607904+0000","last_fullsized":"2024-04-24T00:37:53.607904+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:25.198938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053693430000000004,"stat_sum":{"num_bytes":29758977,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7209,"num_read_kb":84250,"num_write":643,"num_write_kb":47836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":1618865,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]},{"pgid":"3.12","version":"238'1452","reported_seq":5812,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-24T00:37:54.598558+0000","last_change":"2024-04-24T00:37:54.599914+0000","last_active":"2024-04-24T00:37:54.598558+0000","last_peered":"2024-04-24T00:37:54.598558+0000","last_clean":"2024-04-24T00:37:54.598558+0000","last_became_active":"2024-04-24T00:37:52.611238+0000","last_became_peered":"2024-04-24T00:37:52.611238+0000","last_unstale":"2024-04-24T00:37:54.598558+0000","last_undegraded":"2024-04-24T00:37:54.598558+0000","last_fullsized":"2024-04-24T00:37:54.598558+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1452,"log_dups_size":0,"ondisk_log_size":1452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:21:56.940781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037344100000000001,"stat_sum":{"num_bytes":36840311,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7403,"num_read_kb":53792,"num_write":2238,"num_write_kb":43767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"}]},{"pgid":"3.13","version":"232'638","reported_seq":4693,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-04-24T00:37:53.607266+0000","last_change":"2024-04-24T00:37:46.612592+0000","last_active":"2024-04-24T00:37:53.607266+0000","last_peered":"2024-04-24T00:37:53.607266+0000","last_clean":"2024-04-24T00:37:53.607266+0000","last_became_active":"2024-04-24T00:35:28.722941+0000","last_became_peered":"2024-04-24T00:35:28.722941+0000","last_unstale":"2024-04-24T00:37:53.607266+0000","last_undegraded":"2024-04-24T00:37:53.607266+0000","last_fullsized":"2024-04-24T00:37:53.607266+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:28:06.433639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060203920000000003,"stat_sum":{"num_bytes":22578363,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4875,"num_read_kb":39170,"num_write":915,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":962033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"1"},{"start":"52","length":"1"},{"start":"54","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":837133823,"num_objects":506,"num_object_clones":0,"num_object_copies":1012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":181227,"num_read_kb":1541758,"num_write":30020,"num_write_kb":1017899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5442,"num_bytes_recovered":337061731,"num_keys_recovered":493,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26116096,"data_stored":166762378,"data_compressed":9201492,"data_compressed_allocated":19427328,"data_compressed_original":163297893,"omap_allocated":4834,"internal_metadata":0},"log_size":21482,"ondisk_log_size":21482,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":34064,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574904,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80764,"kb_used_data":5160,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291076,"statfs":{"total":96636764160,"available":96554061824,"internally_reserved":0,"allocated":5283840,"data_stored":16849329,"data_compressed":696513,"data_compressed_allocated":1662976,"data_compressed_original":15042905,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74292,"kb_used_data":6992,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297548,"statfs":{"total":96636764160,"available":96560689152,"internally_reserved":0,"allocated":7159808,"data_stored":29150018,"data_compressed":1492846,"data_compressed_allocated":3203072,"data_compressed_original":27176960,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82724,"kb_used_data":7132,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289116,"statfs":{"total":96636764160,"available":96552054784,"internally_reserved":0,"allocated":7303168,"data_stored":31027063,"data_compressed":1372796,"data_compressed_allocated":3211264,"data_compressed_original":28962528,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75040,"kb_used_data":7756,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296800,"statfs":{"total":96636764160,"available":96559923200,"internally_reserved":0,"allocated":7942144,"data_stored":32878848,"data_compressed":1923692,"data_compressed_allocated":3829760,"data_compressed_original":30814570,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57796,"kb_used_data":4408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314044,"statfs":{"total":96636764160,"available":96577581056,"internally_reserved":0,"allocated":4513792,"data_stored":11541922,"data_compressed":567944,"data_compressed_allocated":1175552,"data_compressed_original":9871360,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58184,"kb_used_data":4792,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94313656,"statfs":{"total":96636764160,"available":96577183744,"internally_reserved":0,"allocated":4907008,"data_stored":15667658,"data_compressed":596542,"data_compressed_allocated":1466368,"data_compressed_original":13991936,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893153,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34760,"kb_used_data":7032,"kb_used_omap":6,"kb_used_meta":27705,"kb_avail":94337080,"statfs":{"total":96636764160,"available":96601169920,"internally_reserved":0,"allocated":7200768,"data_stored":28066311,"data_compressed":1654807,"data_compressed_allocated":3252224,"data_compressed_original":26075136,"omap_allocated":6429,"internal_metadata":28370659},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":26029012,"data_compressed":1647822,"data_compressed_allocated":3215360,"data_compressed_original":25485312,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1998848,"data_stored":14226714,"data_compressed":596542,"data_compressed_allocated":1466368,"data_compressed_original":13991936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1605632,"data_stored":10100978,"data_compressed":567944,"data_compressed_allocated":1175552,"data_compressed_original":9871360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5033984,"data_stored":31437904,"data_compressed":1923692,"data_compressed_allocated":3829760,"data_compressed_original":30814570,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4059136,"data_stored":27024800,"data_compressed":1282788,"data_compressed_allocated":2961408,"data_compressed_original":26439392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3313664,"data_stored":19507273,"data_compressed":1220787,"data_compressed_allocated":2433024,"data_compressed_original":19066880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2166784,"data_stored":13485231,"data_compressed":675403,"data_compressed_allocated":1527808,"data_compressed_original":13146457,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:37:58.669 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T00:37:58.669 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:37:58.669 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [1, 2, 3, 4, 6, 5, 0] 2024-04-24T00:37:58.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-04-24T00:37:58.720 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:58.718+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:58.755 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:58.755 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:58.755 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:58.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:37:58.821 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:58.818+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:58.922 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:37:58.918+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:37:58.946 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:58.946 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:58.946 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:58.946 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:58.946 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:58.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:37:59.022 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:37:59.018+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:37:59.051 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:37:59.046+0000 7f841a5f8640 -1 mon.a@0(leader).osd e246 definitely_dead 0 2024-04-24T00:37:59.123 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:59.118+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:59.127 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:59.127 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:59.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:59.127 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 225.90969120599999, 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.128 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:59.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:37:59.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:59.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 211.487905512, 2024-04-24T00:37:59.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:59.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:37:59.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:37:59.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.345777946, 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.139 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:37:59.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:59.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:37:59.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.20800616400001, 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 145.17052384600001, 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.151 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.50817798200001, 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.157 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:37:59.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:37:59.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:37:59.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.49831994600001, 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.163 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:37:59.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:37:59.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.168 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 141.47797033800001, 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:37:59.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:37:59.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 130.443925003, 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:59.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 130.44161516700001, 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.179 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.184 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.49389727499999, 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:37:59.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:37:59.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:37:59.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:37:59.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:37:59.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.604507279, 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.191 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:37:59.196 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.561031482, 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.197 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:37:59.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.58705818, 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.201 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:37:59.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 99.608628723999999, 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.207 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:37:59.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:37:59.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 99.574626301999999, 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.213 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157093 3.13 3:cd0b7234:::rbd_data.115ee709dad1.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.238021066000002, 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12167272799999999, 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.217 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157093 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885996+0000", 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.426e-06 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886003+0000", 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8859999999999998e-06 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886020+0000", 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6135000000000001e-05 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886027+0000", 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3379999999999999e-06 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886099+0000", 2024-04-24T00:37:59.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1802000000000001e-05 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886260+0000", 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016156800000000001 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.895327+0000", 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090669270000000007 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007496+0000", 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112168567 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007520+0000", 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4646000000000001e-05 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007599+0000", 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8424999999999995e-05 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007668+0000", 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9008000000000001e-05 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:157093 3.13 e197/106 ondisk, result = 0)", 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:59.223 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.206603028000004, 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090093925000000005, 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917416+0000", 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917426+0000", 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-04-24T00:37:59.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917465+0000", 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9295999999999998e-05 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917484+0000", 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007420+0000", 2024-04-24T00:37:59.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089936099000000005 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007435+0000", 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4966999999999999e-05 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007507+0000", 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2206000000000006e-05 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169765 3.13 3:c9e19d2e:::rbd_data.115ee24d4bb.0000000000000011:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.264237428000001, 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096474246999999999, 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.228 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169765 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859781+0000", 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859792+0000", 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859819+0000", 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7025999999999998e-05 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859833+0000", 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3763e-05 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859956+0000", 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122989 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.860147+0000", 2024-04-24T00:37:59.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190843 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.866555+0000", 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064084540000000001 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.955923+0000", 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089367567999999994 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956056+0000", 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013292600000000001 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956157+0000", 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101029 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956253+0000", 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5921999999999994e-05 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.233 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:173908 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [delete] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.4551581559999995, 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97469691999999997, 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.234 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 173908 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668860+0000", 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7269999999999999e-06 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668868+0000", 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3799999999999994e-06 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668889+0000", 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0619999999999999e-05 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668906+0000", 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7303e-05 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669040+0000", 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134213 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669057+0000", 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7045999999999999e-05 2024-04-24T00:37:59.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636000+0000", 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96694319299999998 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636222+0000", 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000221411 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636415+0000", 2024-04-24T00:37:59.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193048 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.638630+0000", 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002215276 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.640380+0000", 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017497420000000001 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643297+0000", 2024-04-24T00:37:59.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029172339999999999 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643383+0000", 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5566e-05 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643555+0000", 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017216099999999999 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.240 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.241 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:59.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:37:59.243 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:37:59.238+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:37:59.342 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:59.338+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:59.413 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:59.413 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:37:59.413 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:59.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:37:59.468 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:37:59.466+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:37:59.545 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:37:59.538+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:37:59.592 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:59.593 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:37:59.593 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:37:59.593 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:37:59.593 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:59.596 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.4. 2024-04-24T00:37:59.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:37:59.660 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:59.658+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:37:59.734 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:37:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 212.02319199600001, 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 212.023094071, 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.740 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:37:59.746 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 198.66371453900001, 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:37:59.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:37:59.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 175.25297041300001, 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.753 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:37:59.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:59.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:59.758 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 175.22770180699999, 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:37:59.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:37:59.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:59.762 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 161.37449808700001, 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:59.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 161.341892973, 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.768 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.93731827900001, 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.772 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:37:59.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:37:59.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 160.879478049, 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:37:59.778 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:37:59.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:37:59.782 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.059168149, 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 114.029526096, 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:37:59.788 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:37:59.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:37:59.792 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.883010685, 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:37:59.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:37:59.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.094238127, 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:37:59.799 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:37:59.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:37:59.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 109.071094714, 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:37:59.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:37:59.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:37:59.808 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 99.197785965999998, 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.809 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:37:59.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:37:59.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 99.145103688999995, 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.815 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:37:59.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:37:59.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:166979 3.5 3:a3485662:::rbd_data.115e4ca9eff1.0000000000000010:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.745371736000003, 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.145524811, 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.819 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 166979 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986190+0000", 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.323e-06 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986212+0000", 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1183999999999999e-05 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986246+0000", 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4409999999999998e-05 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986265+0000", 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986437+0000", 2024-04-24T00:37:59.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017130199999999999 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986581+0000", 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014409500000000001 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.997390+0000", 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010809456 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131432+0000", 2024-04-24T00:37:59.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.134041997 2024-04-24T00:37:59.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131521+0000", 2024-04-24T00:37:59.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9024999999999996e-05 2024-04-24T00:37:59.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131625+0000", 2024-04-24T00:37:59.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103923 2024-04-24T00:37:59.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131713+0000", 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7672999999999999e-05 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:166979 3.5 e206/56 ondisk, result = 0)", 2024-04-24T00:37:59.826 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.701836768, 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10200487, 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029726+0000", 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4899999999999999e-06 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029738+0000", 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2483000000000001e-05 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029784+0000", 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5920000000000001e-05 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029809+0000", 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5193000000000001e-05 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131491+0000", 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101681585 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131510+0000", 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9256999999999999e-05 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131728+0000", 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021794199999999999 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169782 3.d 3:b3c6180a:::rbd_data.115ee24d4bb.000000000000000f:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:59.830 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.598714227999999, 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.117826449, 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169782 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132848+0000", 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0770000000000002e-06 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132859+0000", 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:37:59.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132887+0000", 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8070000000000001e-05 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132908+0000", 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.09e-05 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133079+0000", 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017107900000000001 2024-04-24T00:37:59.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133247+0000", 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168138 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.141250+0000", 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080027740000000007 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250223+0000", 2024-04-24T00:37:59.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108972528 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250437+0000", 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021457900000000001 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250540+0000", 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102682 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250672+0000", 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013205899999999999 2024-04-24T00:37:59.835 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169782 3.d e210/106 ondisk, result = 0)", 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 37.569797895999997, 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088930206999999997, 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:37:59.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161764+0000", 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3259999999999998e-06 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:37:59.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161773+0000", 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5329999999999994e-06 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161803+0000", 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9867000000000001e-05 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161825+0000", 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2082999999999998e-05 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250323+0000", 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088498544999999998 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250342+0000", 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8983000000000001e-05 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250692+0000", 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034987000000000001 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.839 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:37:59.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:37:59.840 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:37:59.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:37:59.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:37:59.842 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:37:59.838+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:37:59.942 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:37:59.938+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:00.015 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:00.015 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:00.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.015 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:174837 3.a 3.93d69d6a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-04-24T00:38:00.015 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:59.621320+0000", 2024-04-24T00:38:00.015 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.39435607099999997, 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.39440424699999999, 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 174837 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:59.621320+0000", 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:59.621320+0000", 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:59.621322+0000", 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4020000000000001e-06 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:59.621326+0000", 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4780000000000002e-06 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:59.621338+0000", 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1787000000000001e-05 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:59.621346+0000", 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:59.621406+0000", 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9799999999999997e-05 2024-04-24T00:38:00.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:59.621416+0000", 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0576e-05 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:38:00.019 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:00.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:38:00.073 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:00.070+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:00.145 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:00.138+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:00.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:38:00.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:38:00.277 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:00.274+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.80017488600001, 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.76613783299999, 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.651023837, 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.62357387599999, 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 142.703502044, 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 142.69723078600001, 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.956475021000003, 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 69.935583475000001, 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162926 3.a 3:5391198c:::rbd_data.115e135fe8e7.0000000000000024:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550326706999996, 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069942328999999998, 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162926 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782102+0000", 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782103+0000", 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782107+0000", 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.895e-06 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782118+0000", 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1226999999999999e-05 2024-04-24T00:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782128+0000", 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9909999999999997e-06 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.851774+0000", 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069645631 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.851886+0000", 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111928 2024-04-24T00:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852045+0000", 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015879199999999999 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162938 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 16384~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550287566999998, 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070154141000000003, 2024-04-24T00:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162938 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0500000000000002e-07 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782146+0000", 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0489999999999997e-06 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782158+0000", 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1701e-05 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782169+0000", 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852078+0000", 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069909099000000002 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852143+0000", 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4906999999999994e-05 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852296+0000", 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015313699999999999 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162939 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 49152~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550247470000002, 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070386700999999996, 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162939 2024-04-24T00:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782186+0000", 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.924e-06 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782198+0000", 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2029e-05 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782208+0000", 2024-04-24T00:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9240000000000002e-06 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852330+0000", 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070121292000000002 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852410+0000", 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9945999999999997e-05 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852568+0000", 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158763 2024-04-24T00:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162940 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550207481000001, 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070748154999999993, 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162940 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782226+0000", 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782239+0000", 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2516e-05 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782249+0000", 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064000000000001e-05 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852602+0000", 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070352985000000007 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852685+0000", 2024-04-24T00:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3584999999999993e-05 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852970+0000", 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284412 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162941 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 114688~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550166648000001, 2024-04-24T00:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071218216000000001, 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162941 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782263+0000", 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782267+0000", 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.172e-06 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782280+0000", 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2308e-05 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782290+0000", 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0674e-05 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853061+0000", 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070770437000000005 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853196+0000", 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013527100000000001 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853481+0000", 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028453700000000002 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162942 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 147456~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550125143999999, 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071662612000000001, 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162942 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782305+0000", 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3399999999999998e-07 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782308+0000", 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5480000000000002e-06 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782321+0000", 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2814e-05 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782331+0000", 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0363999999999999e-05 2024-04-24T00:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853550+0000", 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071219062999999999 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853680+0000", 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129219 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853967+0000", 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028687 2024-04-24T00:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162943 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 180224~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550083979999997, 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071922708000000002, 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162943 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782346+0000", 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6300000000000004e-07 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782350+0000", 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9129999999999996e-06 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782361+0000", 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1249e-05 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782370+0000", 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0829999999999993e-06 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854021+0000", 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071651144 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854119+0000", 2024-04-24T00:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7157999999999994e-05 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854268+0000", 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014929799999999999 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162944 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 212992~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550046235000003, 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072204638000000002, 2024-04-24T00:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162944 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782384+0000", 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7800000000000001e-07 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782387+0000", 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7720000000000001e-06 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782399+0000", 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1508000000000001e-05 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782409+0000", 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7550000000000006e-06 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854321+0000", 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071912767000000002 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854437+0000", 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115282 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854587+0000", 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015087600000000001 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162945 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 245760~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.550007450999999, 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072460953999999994, 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162945 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782427+0000", 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6210000000000003e-06 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782447+0000", 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0129999999999999e-05 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782457+0000", 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.03e-05 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854619+0000", 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072161259000000005 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854739+0000", 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120542 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854883+0000", 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143354 2024-04-24T00:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162946 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 278528~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.549958468, 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072673115999999996, 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162946 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782493+0000", 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1645999999999999e-05 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782503+0000", 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4509999999999998e-06 2024-04-24T00:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782511+0000", 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854939+0000", 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072427668000000001 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855054+0000", 2024-04-24T00:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114957 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855144+0000", 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0175999999999999e-05 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162947 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 311296~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.549908659000003, 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072916576999999996, 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162947 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782521+0000", 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782524+0000", 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.971e-06 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782532+0000", 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7940000000000003e-06 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782538+0000", 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855191+0000", 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072653180999999997 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855292+0000", 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100948 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855437+0000", 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014472700000000001 2024-04-24T00:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162948 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 47.549857447999997, 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073391995000000002, 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162948 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782573+0000", 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9700000000000005e-07 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782577+0000", 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782587+0000", 2024-04-24T00:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5140000000000007e-06 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782594+0000", 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.968e-06 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855774+0000", 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073180178999999998 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855862+0000", 2024-04-24T00:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7793999999999993e-05 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855964+0000", 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010183099999999999 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:00.427 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:00.428 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:00.428 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:00.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:00.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:38:00.429 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:00.422+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:00.430 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:00.430 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:00.431 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:00.431 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:00.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:00.452 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:00.473 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:00.473 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:00.494 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:00.494 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:00.514 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:00.515 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:00.528 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:00.522+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:00.605 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:38:00.607 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:00.607 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:00.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:38:00.629 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:00.626+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:00.731 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:00.726+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:00.748 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:00.748 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:00.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:38:00.832 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:00.826+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:00.885 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:00.885 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:00.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:38:00.933 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:00.930+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:01.034 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:01.030+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:01.036 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:01.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:38:01.137 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:01.134+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:01.222 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:01.222 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:01.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:38:01.237 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:01.234+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:01.337 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:01.334+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:01.407 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:01.407 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:01.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:38:01.465 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:01.462+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:01.539 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:01.534+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:01.541 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:01.541 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:01.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:38:01.646 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:01.642+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:01.740 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:01.738+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:01.769 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:01.770 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:01.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:38:01.881 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:01.878+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:01.943 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:01.938+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:01.982 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:01.982 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:01.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:38:02.043 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:02.038+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:02.144 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:02.138+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:02.191 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:02.191 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:02.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:38:02.253 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:02.250+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:02.346 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:02.342+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:02.407 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:02.407 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:02.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:38:02.469 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:02.466+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:02.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:38:02.547 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:02.542+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:02.618 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:02.619 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:02.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:38:02.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:38:02.685 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:02.682+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:02.714 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:02.737 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:02.749 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:02.746+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:02.761 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:02.761 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:02.791 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:02.821 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:02.821 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:02.839 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:02.840 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:38:02.840 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:38:02.840 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:02.850 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:02.846+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:02.857 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:02.857 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:02.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:38:02.881 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:02.881 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:02.918 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:02.918 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:02.989 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:02.986+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:03.017 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:38:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:38:03.017 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:38:03.017 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:03.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:38:03.077 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:03.074+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:03.153 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:03.146+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:03.212 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.99838876199999, 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:38:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.218 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.219 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:38:03.219 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.99803784700001, 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:38:03.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:38:03.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.991149527, 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 178.73286790099999, 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:38:03.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:03.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:38:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 178.72259148500001, 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:38:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.260640682, 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:38:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.23460666099999, 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.020663754, 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.01065181000001, 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:38:03.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:38:03.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:38:03.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.737272948, 2024-04-24T00:38:03.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:38:03.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:38:03.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:38:03.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:38:03.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:89272 3.15 e119/106 ondisk, result = 0)", 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.72060449899999, 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047220941000000002, 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489545+0000", 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489547+0000", 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489554+0000", 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1999999999999997e-06 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489584+0000", 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0482999999999999e-05 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.489599+0000", 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4476e-05 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536717+0000", 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047118321999999997 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536737+0000", 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9933e-05 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536766+0000", 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.879e-05 2024-04-24T00:38:03.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 134.53470671599999, 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:38:03.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:38:03.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.850948359, 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:38:03.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:38:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:38:03.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:38:03.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.828731361, 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:38:03.278 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.524161523, 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.51153656300001, 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:38:03.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.162045075999998, 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.292 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:38:03.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:38:03.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.147026881000002, 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:03.298 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:38:03.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:38:03.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.254210477999997, 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.302 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:38:03.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.307 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.216858451, 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:03.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:38:03.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.312 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.312 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.312 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:03.312 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:03.312 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:03.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:38:03.314 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:03.310+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:03.415 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:03.410+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:03.446 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:03.446 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:03.446 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:03.446 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:03.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:38:03.516 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:03.510+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:03.592 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:03.592 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:03.592 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:03.592 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:03.592 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:03.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:38:03.641 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:38:03.652 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:03.650+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:03.718 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:03.714+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:03.749 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 230.531925591, 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.750 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:38:03.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:38:03.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 216.11013989700001, 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.756 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:03.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 154.96801233100001, 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.761 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:38:03.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:38:03.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:38:03.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 149.830240549, 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 149.79275823099999, 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:03.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.13041236699999, 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.779 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:38:03.784 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.12055433099999, 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.785 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:38:03.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:38:03.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.10020472299999, 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.791 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:38:03.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:38:03.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 135.06615938799999, 2024-04-24T00:38:03.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:38:03.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:38:03.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 135.06384955199999, 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.801 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:38:03.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:38:03.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:03.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:03.806 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 131.11613166000001, 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:03.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:38:03.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.226741664, 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.813 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:38:03.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.818 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:38:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.183265867, 2024-04-24T00:38:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:38:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.819 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:38:03.824 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.209292565, 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.230863109, 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.830 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:38:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:38:03.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:38:03.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:38:03.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 104.196860687, 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:38:03.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:38:03.839 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157093 3.13 3:cd0b7234:::rbd_data.115ee709dad1.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.860255451, 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12167272799999999, 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.840 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157093 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885996+0000", 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.426e-06 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886003+0000", 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8859999999999998e-06 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886020+0000", 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6135000000000001e-05 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886027+0000", 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3379999999999999e-06 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886099+0000", 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1802000000000001e-05 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886260+0000", 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016156800000000001 2024-04-24T00:38:03.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.895327+0000", 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090669270000000007 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007496+0000", 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112168567 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007520+0000", 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4646000000000001e-05 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007599+0000", 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8424999999999995e-05 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007668+0000", 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9008000000000001e-05 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.845 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:157093 3.13 e197/106 ondisk, result = 0)", 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.828837413000002, 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090093925000000005, 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917416+0000", 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917426+0000", 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917465+0000", 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9295999999999998e-05 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917484+0000", 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007420+0000", 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089936099000000005 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007435+0000", 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4966999999999999e-05 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007507+0000", 2024-04-24T00:38:03.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2206000000000006e-05 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169765 3.13 3:c9e19d2e:::rbd_data.115ee24d4bb.0000000000000011:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 41.886471813, 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096474246999999999, 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169765 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859781+0000", 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859792+0000", 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859819+0000", 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7025999999999998e-05 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859833+0000", 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3763e-05 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859956+0000", 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122989 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.860147+0000", 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190843 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.866555+0000", 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064084540000000001 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.955923+0000", 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089367567999999994 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956056+0000", 2024-04-24T00:38:03.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013292600000000001 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956157+0000", 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101029 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956253+0000", 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5921999999999994e-05 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:173908 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [delete] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 14.077392541, 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97469691999999997, 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 173908 2024-04-24T00:38:03.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:03.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668860+0000", 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7269999999999999e-06 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668868+0000", 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3799999999999994e-06 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668889+0000", 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0619999999999999e-05 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668906+0000", 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7303e-05 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669040+0000", 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134213 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669057+0000", 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7045999999999999e-05 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636000+0000", 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96694319299999998 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636222+0000", 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000221411 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636415+0000", 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193048 2024-04-24T00:38:03.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.638630+0000", 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002215276 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.640380+0000", 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017497420000000001 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643297+0000", 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029172339999999999 2024-04-24T00:38:03.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643383+0000", 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5566e-05 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643555+0000", 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017216099999999999 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:03.863 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:03.863 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:03.865 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:38:03.865 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:03.858+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:03.964 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:03.962+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:04.039 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:04.040 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:04.040 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:04.040 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:04.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:38:04.065 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:04.062+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:04.166 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:04.162+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:04.220 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:04.221 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:04.221 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:04.221 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:04.221 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:04.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:38:04.276 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:04.274+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:04.368 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:04.366+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:04.394 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:04.394 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:04.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 216.68496682599999, 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:38:04.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:38:04.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:38:04.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:38:04.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 216.68486890099999, 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:38:04.400 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:04.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:04.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:38:04.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:38:04.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:38:04.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:38:04.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:38:04.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:38:04.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:38:04.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:38:04.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:38:04.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:38:04.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:04.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:38:04.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:38:04.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:38:04.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:38:04.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:38:04.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:38:04.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:38:04.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:38:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:38:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:38:04.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:38:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:38:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:38:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:38:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:38:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:38:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:38:04.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:38:04.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.412 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.413 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:38:04.413 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:04.413 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 203.325489369, 2024-04-24T00:38:04.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:38:04.413 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.413 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.413 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.414 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.414 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:38:04.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.414 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:38:04.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:38:04.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:38:04.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:38:04.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:38:04.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:38:04.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 179.914745243, 2024-04-24T00:38:04.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:38:04.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:38:04.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:38:04.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:38:04.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:38:04.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:38:04.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:38:04.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:38:04.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:38:04.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.428 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.428 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:38:04.428 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:04.428 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 179.889476637, 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:04.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:38:04.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:38:04.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:38:04.435 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 166.03627291699999, 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:04.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:38:04.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.441 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 166.00366780300001, 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:38:04.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:38:04.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.445 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 165.59909310899999, 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.446 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:38:04.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.451 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 165.54125287900001, 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:38:04.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:38:04.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.720942979, 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.456 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:04.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:38:04.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:38:04.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.691300926, 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.462 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:38:04.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:38:04.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:38:04.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:38:04.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:38:04.466 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.544785515, 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.467 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:38:04.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:38:04.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.756012957, 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:38:04.474 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:38:04.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:38:04.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:04.480 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 113.732869544, 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:04.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:38:04.484 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 103.859560796, 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:04.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:38:04.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:38:04.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:04.490 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 103.80687851899999, 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:04.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:166979 3.5 3:a3485662:::rbd_data.115e4ca9eff1.0000000000000010:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.407146566000002, 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.145524811, 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 166979 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986190+0000", 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.323e-06 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986212+0000", 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1183999999999999e-05 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986246+0000", 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4409999999999998e-05 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986265+0000", 2024-04-24T00:38:04.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986437+0000", 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017130199999999999 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986581+0000", 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014409500000000001 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.997390+0000", 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010809456 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131432+0000", 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.134041997 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131521+0000", 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9024999999999996e-05 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131625+0000", 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103923 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131713+0000", 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7672999999999999e-05 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:166979 3.5 e206/56 ondisk, result = 0)", 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.363611597999999, 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10200487, 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029726+0000", 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4899999999999999e-06 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029738+0000", 2024-04-24T00:38:04.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2483000000000001e-05 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029784+0000", 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5920000000000001e-05 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029809+0000", 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5193000000000001e-05 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131491+0000", 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101681585 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131510+0000", 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9256999999999999e-05 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131728+0000", 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021794199999999999 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.504 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169782 3.d 3:b3c6180a:::rbd_data.115ee24d4bb.000000000000000f:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.260489057999997, 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.117826449, 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169782 2024-04-24T00:38:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132848+0000", 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0770000000000002e-06 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132859+0000", 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132887+0000", 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8070000000000001e-05 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132908+0000", 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.09e-05 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133079+0000", 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017107900000000001 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133247+0000", 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168138 2024-04-24T00:38:04.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.141250+0000", 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080027740000000007 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250223+0000", 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108972528 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250437+0000", 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021457900000000001 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250540+0000", 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102682 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250672+0000", 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013205899999999999 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169782 3.d e210/106 ondisk, result = 0)", 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 42.231572726000003, 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088930206999999997, 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:04.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161764+0000", 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3259999999999998e-06 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161773+0000", 2024-04-24T00:38:04.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5329999999999994e-06 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161803+0000", 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9867000000000001e-05 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161825+0000", 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2082999999999998e-05 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250323+0000", 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088498544999999998 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250342+0000", 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8983000000000001e-05 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250692+0000", 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034987000000000001 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.514 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.515 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.515 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:04.516 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:38:04.517 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:04.514+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:04.618 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:04.610+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:04.631 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 6, 5, 0] out_osds: [7, 4] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:38:04.631 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:38:04.631 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-24T00:38:04.631 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:38:04.632 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:38:04.632 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-24T00:38:04.704 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:04.704 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:04.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.704 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:175134 3.a 3.fa1caca (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e252)", 2024-04-24T00:38:04.704 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:38:04.693472+0000", 2024-04-24T00:38:04.704 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.011168137999999999, 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011218518, 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for peered", 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 175134 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.693472+0000", 2024-04-24T00:38:04.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.693472+0000", 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.693476+0000", 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6660000000000001e-06 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.693490+0000", 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3507e-05 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.693534+0000", 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4687000000000001e-05 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.693556+0000", 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1806000000000001e-05 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.693665+0000", 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109218 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.693675+0000", 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.825e-06 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 1 2024-04-24T00:38:04.708 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:04.717 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:04.714+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:04.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:38:04.755 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:04.755 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "" 2024-04-24T00:38:04.755 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:04.818 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:04.814+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:04.846 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:04.846 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:04.847 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:04.847 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:04.847 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:04.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:38:04.919 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:04.914+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:04.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:38:04.981 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 226.44642651500001, 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.982 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.983 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:38:04.984 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:38:04.985 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.986 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.987 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 226.41238946199999, 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.988 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:38:04.989 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:38:04.990 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.991 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 194.297275466, 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.992 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.993 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:38:04.994 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:38:04.995 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:04.996 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 194.269825505, 2024-04-24T00:38:04.997 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:04.998 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:04.999 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.000 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:38:05.001 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 147.34975367300001, 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.002 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.003 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.004 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.005 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 147.34348241500001, 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.006 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.007 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:38:05.008 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:38:05.009 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:38:05.010 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.011 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.602726649999994, 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.012 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:38:05.013 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.014 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.015 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:38:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.017 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.581835104000007, 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.018 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.019 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.020 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.021 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162938 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 16384~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196539196000003, 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070154141000000003, 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162938 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.022 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.023 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0500000000000002e-07 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782146+0000", 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0489999999999997e-06 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782158+0000", 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1701e-05 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.024 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782169+0000", 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852078+0000", 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069909099000000002 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852143+0000", 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4906999999999994e-05 2024-04-24T00:38:05.025 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852296+0000", 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015313699999999999 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162939 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 49152~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:05.026 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196499099, 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070386700999999996, 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162939 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782186+0000", 2024-04-24T00:38:05.028 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.924e-06 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782198+0000", 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2029e-05 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782208+0000", 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9240000000000002e-06 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852330+0000", 2024-04-24T00:38:05.029 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070121292000000002 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852410+0000", 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9945999999999997e-05 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852568+0000", 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158763 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.030 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162940 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196459109999999, 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070748154999999993, 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162940 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:05.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782226+0000", 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782239+0000", 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2516e-05 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782249+0000", 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064000000000001e-05 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852602+0000", 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070352985000000007 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852685+0000", 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3584999999999993e-05 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852970+0000", 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284412 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.034 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162941 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 114688~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196418276999999, 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071218216000000001, 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162941 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.035 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782263+0000", 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:05.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782267+0000", 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.172e-06 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782280+0000", 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2308e-05 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782290+0000", 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0674e-05 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853061+0000", 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070770437000000005 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853196+0000", 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013527100000000001 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853481+0000", 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028453700000000002 2024-04-24T00:38:05.038 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162942 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 147456~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196376772999997, 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071662612000000001, 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162942 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782305+0000", 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3399999999999998e-07 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782308+0000", 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5480000000000002e-06 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782321+0000", 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2814e-05 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782331+0000", 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0363999999999999e-05 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853550+0000", 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071219062999999999 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853680+0000", 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129219 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853967+0000", 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028687 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162943 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 180224~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196335609000002, 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071922708000000002, 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.043 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.045 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.045 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162943 2024-04-24T00:38:05.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.045 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782346+0000", 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6300000000000004e-07 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782350+0000", 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9129999999999996e-06 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782361+0000", 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1249e-05 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782370+0000", 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0829999999999993e-06 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854021+0000", 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071651144 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854119+0000", 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7157999999999994e-05 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854268+0000", 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014929799999999999 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162944 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 212992~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196297864000002, 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072204638000000002, 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162944 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782384+0000", 2024-04-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7800000000000001e-07 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782387+0000", 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7720000000000001e-06 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782399+0000", 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1508000000000001e-05 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782409+0000", 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7550000000000006e-06 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854321+0000", 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071912767000000002 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854437+0000", 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115282 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854587+0000", 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015087600000000001 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162945 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 245760~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196259079999997, 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072460953999999994, 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.053 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162945 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782427+0000", 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6210000000000003e-06 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782447+0000", 2024-04-24T00:38:05.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0129999999999999e-05 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782457+0000", 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.03e-05 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854619+0000", 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072161259000000005 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854739+0000", 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120542 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854883+0000", 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143354 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162946 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 278528~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:05.057 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196210096999998, 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072673115999999996, 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162946 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782493+0000", 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1645999999999999e-05 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782503+0000", 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4509999999999998e-06 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782511+0000", 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854939+0000", 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072427668000000001 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855054+0000", 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114957 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855144+0000", 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0175999999999999e-05 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162947 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 311296~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196160288000002, 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072916576999999996, 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162947 2024-04-24T00:38:05.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782521+0000", 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782524+0000", 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.971e-06 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782532+0000", 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7940000000000003e-06 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782538+0000", 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855191+0000", 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072653180999999997 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855292+0000", 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100948 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855437+0000", 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014472700000000001 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162948 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 52.196109077000003, 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073391995000000002, 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.066 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162948 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782573+0000", 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9700000000000005e-07 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782577+0000", 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782587+0000", 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5140000000000007e-06 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782594+0000", 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.968e-06 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855774+0000", 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073180178999999998 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855862+0000", 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7793999999999993e-05 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855964+0000", 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010183099999999999 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:175102 3.11 3:8ca8b224:::rbd_header.115ec7ed39de:head [watch reconnect cookie 140203704658000 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-24T00:38:05.070 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 1.3436198830000001, 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98633510999999996, 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 175102 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:05.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635063+0000", 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.798e-06 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635070+0000", 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4719999999999998e-06 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635088+0000", 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7853e-05 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635099+0000", 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1058e-05 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635160+0000", 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0868000000000001e-05 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635172+0000", 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2459e-05 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.615851+0000", 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98067890000000002 2024-04-24T00:38:05.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.616075+0000", 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022393399999999999 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.621396+0000", 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005320768 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:05.075 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:05.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:38:05.077 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:05.074+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:05.106 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:05.106 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:05.122 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:05.123 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:05.139 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:05.157 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:05.176 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:05.176 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:05.177 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:05.174+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:05.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:05.195 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:05.214 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:05.214 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:05.233 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:05.234 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:05.250 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:05.250 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:05.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:38:05.309 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:05.306+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:05.379 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:05.374+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:05.426 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:05.426 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:05.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:38:05.480 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:05.478+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:05.581 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:05.578+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:05.613 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:05.613 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:05.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:38:05.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:38:05.717 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:05.714+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:05.765 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:05.766 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:05.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:38:05.821 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:05.818+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:05.883 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:05.878+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:05.936 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:05.936 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:05.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:38:05.993 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:05.990+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:06.085 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:06.082+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:06.113 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:06.114 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:06.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:38:06.186 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:06.186+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:06.287 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:06.286+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:06.293 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:06.293 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:06.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:38:06.388 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:06.386+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:06.431 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:06.431 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:06.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:38:06.489 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:06.486+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:06.572 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:06.572 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:06.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:38:06.629 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:06.626+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:06.672 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:38:06.691 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:06.686+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:06.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:38:06.792 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:06.790+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:06.893 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:06.890+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:06.922 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:06.922 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:06.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:38:06.994 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:06.990+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:07.059 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:07.059 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:07.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:38:07.117 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:07.114+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:07.193 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:07.193 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:07.195 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:07.190+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:07.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:38:07.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:38:07.297 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:07.294+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:07.378 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:07.378 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:07.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:38:07.437 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:07.434+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:07.440 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:07.440 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:07.457 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:07.457 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:07.474 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:07.491 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:07.491 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:07.498 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:07.494+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:07.512 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:07.512 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:07.514 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:07.514 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:38:07.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:38:07.514 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:38:07.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.514 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10258 3.1e 3:7c238a0e:::rbd_data.115e380d621c.0000000000000072:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:38:07.514 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:38:07.514 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 234.299806026, 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064766211000000004, 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10258 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:38:07.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211761+0000", 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211762+0000", 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.099e-06 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211768+0000", 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.489e-06 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211785+0000", 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7169000000000001e-05 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211793+0000", 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5140000000000004e-06 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.211880+0000", 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510999999999999e-05 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212102+0000", 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022236700000000001 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.214199+0000", 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020970049999999999 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276185+0000", 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061985927000000003 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276327+0000", 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014173500000000001 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276430+0000", 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010383 2024-04-24T00:38:07.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276527+0000", 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6564999999999994e-05 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:10259 3.1e 3:7bb3a521:::rbd_data.115e380d621c.0000000000000073:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 234.29945511099999, 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064628339000000007, 2024-04-24T00:38:07.520 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 10259 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212112+0000", 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212119+0000", 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3260000000000001e-06 2024-04-24T00:38:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212129+0000", 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0178e-05 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.212135+0000", 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9349999999999999e-06 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259554+0000", 2024-04-24T00:38:07.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419080000000002 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.259792+0000", 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023835699999999999 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.261561+0000", 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017684689999999999 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276293+0000", 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014732614 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276541+0000", 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024775600000000002 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276638+0000", 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7280000000000004e-05 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276740+0000", 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101504 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:10258 3.1e e40/21 ondisk, result = 0)", 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 234.29256679100001, 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057200001, 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219000+0000", 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219001+0000", 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0449999999999999e-06 2024-04-24T00:38:07.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219007+0000", 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.614e-06 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219022+0000", 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5237e-05 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.219031+0000", 2024-04-24T00:38:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6980000000000004e-06 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276159+0000", 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057128103999999999 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276174+0000", 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4962e-05 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:34:13.276200+0000", 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6341000000000001e-05 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:65821 3.8 3:17a3db5d:::rbd_data.115e8561432.0000000000000102:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.034285165, 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064354003000000007, 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.530 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 65821 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477282+0000", 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477283+0000", 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6249999999999999e-06 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477292+0000", 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.551e-06 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477309+0000", 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7081000000000001e-05 2024-04-24T00:38:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477316+0000", 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6290000000000001e-06 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477396+0000", 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9332000000000005e-05 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.477508+0000", 2024-04-24T00:38:07.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112487 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.481131+0000", 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036224949999999999 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541424+0000", 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060293411999999998 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541456+0000", 2024-04-24T00:38:07.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1442999999999997e-05 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541549+0000", 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3176000000000004e-05 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541636+0000", 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6772000000000002e-05 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:65821 3.8 e85/21 ondisk, result = 0)", 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.02400874899999, 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053881200999999997, 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487558+0000", 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487559+0000", 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.540000000000001e-07 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487565+0000", 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8370000000000002e-06 2024-04-24T00:38:07.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487577+0000", 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.487584+0000", 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4479999999999999e-06 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541331+0000", 2024-04-24T00:38:07.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053747534 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541351+0000", 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9573999999999999e-05 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:04.541439+0000", 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8383999999999999e-05 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:81507 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.56205794600001, 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058983524000000002, 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81507 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949509+0000", 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.949511+0000", 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954544+0000", 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050330549999999998 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954579+0000", 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4774999999999998e-05 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954594+0000", 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.47e-05 2024-04-24T00:38:07.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954705+0000", 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110887 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.954887+0000", 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018265999999999999 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.955031+0000", 2024-04-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014364 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.004684+0000", 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049652741 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008406+0000", 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037227459999999999 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008454+0000", 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7549999999999997e-05 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.008492+0000", 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8324000000000001e-05 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:81512 3.5 e111/56 3:a10df431:::rbd_data.115e6f07caf6.0000000000000005:head v 111'230, mlcod=98'228)", 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.53602392499999, 2024-04-24T00:38:07.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048486027000000001, 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975543+0000", 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.975544+0000", 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982206+0000", 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066624079999999999 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982226+0000", 2024-04-24T00:38:07.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9508e-05 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982236+0000", 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0225e-05 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982333+0000", 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6881999999999998e-05 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:33.982352+0000", 2024-04-24T00:38:07.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8759000000000001e-05 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.023799+0000", 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041447020000000001 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.024029+0000", 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022997499999999999 2024-04-24T00:38:07.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84137 3.1 e111/59 3:8281a7ce:::rbd_data.115e6f07caf6.0000000000000012:head v 111'182, mlcod=108'180)", 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.32208101800001, 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066452516000000003, 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189486+0000", 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.189487+0000", 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.327e-06 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211264+0000", 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021777350000000001 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211280+0000", 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5187999999999999e-05 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211288+0000", 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1009999999999992e-06 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211354+0000", 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5993000000000003e-05 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.211376+0000", 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1953000000000001e-05 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255810+0000", 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044434803000000002 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.255938+0000", 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012780100000000001 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:84136 3.12 e111/106 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head v 111'967, mlcod=108'965)", 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.31206907399999, 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058649185, 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199498+0000", 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.199499+0000", 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5060000000000001e-06 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201200+0000", 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017010899999999999 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201258+0000", 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8217999999999997e-05 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201270+0000", 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1936999999999999e-05 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201365+0000", 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4658000000000001e-05 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.201387+0000", 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1515999999999999e-05 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258107+0000", 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0567205 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:34.258147+0000", 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9759999999999999e-05 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.558 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89272 3.15 3:af32ebe0:::rbd_data.115e6713a289.0000000000000076:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.03869021200001, 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064096522000000003, 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89272 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472877+0000", 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472879+0000", 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1619999999999998e-06 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472886+0000", 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.542e-06 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472904+0000", 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7836999999999999e-05 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.472912+0000", 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1329999999999999e-06 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473006+0000", 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3980000000000005e-05 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.473134+0000", 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127346 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.477612+0000", 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044784769999999998 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536748+0000", 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059136286000000003 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536784+0000", 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5539e-05 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536887+0000", 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102772 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:43.536973+0000", 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6447999999999996e-05 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:89922 3.19 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head [write 0~5242880 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 138.83612398, 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.084179734000000006, 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 89922 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675443+0000", 2024-04-24T00:38:07.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.675446+0000", 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1650000000000002e-06 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678876+0000", 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034297709999999999 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678911+0000", 2024-04-24T00:38:07.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5178000000000001e-05 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.678924+0000", 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3553000000000001e-05 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679005+0000", 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1007000000000006e-05 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679180+0000", 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174573 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.679343+0000", 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016261699999999999 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.714600+0000", 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035256943999999998 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759478+0000", 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044878553000000002 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759561+0000", 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3145000000000004e-05 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:35:48.759623+0000", 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1228000000000004e-05 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:91798 3.15 3:aea1436d:::rbd_data.115ee57427d8.000000000000000b:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.15236562299999, 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11196292300000001, 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 91798 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359201+0000", 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359202+0000", 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2024-04-24T00:38:07.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359209+0000", 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5660000000000003e-06 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359226+0000", 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359236+0000", 2024-04-24T00:38:07.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0224000000000001e-05 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359293+0000", 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6360999999999998e-05 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.359356+0000", 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3009999999999995e-05 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.364524+0000", 2024-04-24T00:38:07.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005168326 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470901+0000", 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106377057 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470981+0000", 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9406999999999994e-05 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471081+0000", 2024-04-24T00:38:07.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100083 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471164+0000", 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3325999999999995e-05 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:91798 3.15 e138/106 ondisk, result = 0)", 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.130148625, 2024-04-24T00:38:07.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089756738000000003, 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381418+0000", 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381420+0000", 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381426+0000", 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7449999999999998e-06 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381452+0000", 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5792999999999999e-05 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.381465+0000", 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.323e-05 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470951+0000", 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.089485568000000001 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.470968+0000", 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7396000000000001e-05 2024-04-24T00:38:07.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:02.471175+0000", 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020662600000000001 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:105170 3.16 3:6a24543b:::rbd_data.115ec2448e7a.0000000000000002:head [write 0~393216 [fadvise_sequential+fadvise_nocache] in=393216b,zero 393216~393216,write 786432~131072 [fadvise_sequential+fadvise_nocache] in=131072b,write 1048576~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 1966080~393216,write 2359296~131072 [fadvise_sequential+fadvise_nocache] in=131072b,zero 2490368~131072,write 2621440~917504 [fadvise_sequential+fadvise_nocache] in=917504b,zero 3538944~393216,write 3932160~131072 [fadvise_sequential+fadvise_nocache] in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.825578787, 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047961190000000001, 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105170 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685988+0000", 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.685990+0000", 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2840000000000001e-06 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688426+0000", 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024353460000000001 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688459+0000", 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3254999999999999e-05 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688485+0000", 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6262999999999999e-05 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688579+0000", 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4248999999999998e-05 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688721+0000", 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141984 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.688960+0000", 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023815199999999999 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.711643+0000", 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022683596 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733777+0000", 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022133547 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733873+0000", 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6039000000000007e-05 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.733949+0000", 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6475000000000002e-05 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.584 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4446.0:105171 3.1e e152/44 3:78e05c73:::rbd_data.115ec2448e7a.0000000000000001:head v 152'445, mlcod=150'443)", 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.812953827, 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052713442999999999, 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698613+0000", 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.698614+0000", 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706687+0000", 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080729919999999993 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706731+0000", 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3639e-05 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706741+0000", 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5810000000000003e-06 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706819+0000", 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7952000000000002e-05 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.706855+0000", 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751283+0000", 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044428128999999997 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:15.751326+0000", 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3735000000000001e-05 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:157101 3.16 3:6dae1b85:::rbd_data.115ee709dad1.000000000000007f:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.46346234, 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076023489999999999, 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.589 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 157101 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048104+0000", 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048107+0000", 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048118+0000", 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1216999999999999e-05 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048146+0000", 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7962000000000001e-05 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048160+0000", 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4087000000000001e-05 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048269+0000", 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010873499999999999 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.048500+0000", 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023125399999999999 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.052340+0000", 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003840268 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123914+0000", 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071573322999999994 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123948+0000", 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4406000000000002e-05 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124046+0000", 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8082e-05 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.124128+0000", 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1846000000000002e-05 2024-04-24T00:38:07.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:157101 3.16 e197/112 ondisk, result = 0)", 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.448444145000003, 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060809397000000001, 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063123+0000", 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063124+0000", 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.75e-06 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063133+0000", 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4670000000000004e-06 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063161+0000", 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8333000000000001e-05 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.063176+0000", 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4836e-05 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123836+0000", 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060660432 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123857+0000", 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0627000000000001e-05 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:07.123932+0000", 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4951999999999997e-05 2024-04-24T00:38:07.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:169766 3.15 3:a876155d:::rbd_data.115ee24d4bb.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.555627741999999, 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14666620899999999, 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 169766 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955939+0000", 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955942+0000", 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3330000000000001e-06 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955958+0000", 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5778999999999999e-05 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.955992+0000", 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4189000000000001e-05 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956009+0000", 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956146+0000", 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136933 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.956310+0000", 2024-04-24T00:38:07.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000164606 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.966523+0000", 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010212723999999999 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102341+0000", 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13581758099999999 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102423+0000", 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2233999999999999e-05 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102523+0000", 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010028500000000001 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102605+0000", 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1917999999999997e-05 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4446.0:169766 3.15 e210/106 ondisk, result = 0)", 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.518275715000001, 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109333741, 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993291+0000", 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993293+0000", 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993304+0000", 2024-04-24T00:38:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366e-05 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993335+0000", 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1723000000000001e-05 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:21.993353+0000", 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7096e-05 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102405+0000", 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109052677 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102415+0000", 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:22.102625+0000", 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020949799999999999 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:175101 3.15 3:a9df8c85:::rbd_data.115ec7ed39de.0000000000000009:head [delete] snapc 1d=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:38:02.850996+0000", 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6605703170000004, 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.80166834600000003, 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 175101 2024-04-24T00:38:07.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:02.850996+0000", 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:02.850996+0000", 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:02.850997+0000", 2024-04-24T00:38:07.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1100000000000005e-07 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:02.851002+0000", 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2699999999999998e-06 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:02.851011+0000", 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3309999999999993e-06 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:02.851018+0000", 2024-04-24T00:38:07.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0339999999999999e-06 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:02.851080+0000", 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2241000000000003e-05 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:02.851091+0000", 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0627e-05 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.647562+0000", 2024-04-24T00:38:07.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.79647123500000006 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.647734+0000", 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017196299999999999 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.647876+0000", 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142526 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.650688+0000", 2024-04-24T00:38:07.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028111339999999999 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.652502+0000", 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018143790000000001 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.652586+0000", 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3955999999999994e-05 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:07.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.652665+0000", 2024-04-24T00:38:07.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8838999999999995e-05 2024-04-24T00:38:07.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.615 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:07.615 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:07.615 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:07.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:38:07.617 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:07.617 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:07.617 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:07.618 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:07.618 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:07.619 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:07.619 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:07.610+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:07.718 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:07.714+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:07.759 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:07.759 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:07.759 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:07.759 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:07.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:38:07.820 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:07.818+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:07.901 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:07.902 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:07.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:38:07.964 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:07.962+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:08.020 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:08.014+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:08.092 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:08.092 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 234.87469482399999, 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.093 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:38:08.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.45290912999999, 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:38:08.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 159.310781564, 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.104 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:08.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:38:08.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:38:08.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:38:08.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:38:08.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:38:08.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:38:08.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 154.17300978200001, 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.113 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:38:08.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 154.13552746400001, 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.119 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:08.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:38:08.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:38:08.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.4731816, 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:38:08.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:38:08.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.130 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.46332356400001, 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.131 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:38:08.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:38:08.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.442973956, 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:38:08.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:38:08.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:38:08.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 139.408928621, 2024-04-24T00:38:08.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:38:08.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 139.40661878500001, 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.147 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:38:08.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:38:08.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:38:08.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 135.45890089299999, 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:38:08.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:38:08.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:38:08.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.569510897, 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.159 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:08.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:38:08.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.5260351, 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 117.552061798, 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.169 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:08.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.573632342, 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:38:08.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.53962992, 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:38:08.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157093 3.13 3:cd0b7234:::rbd_data.115ee709dad1.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.203024683999999, 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12167272799999999, 2024-04-24T00:38:08.184 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157093 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885996+0000", 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.426e-06 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886003+0000", 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8859999999999998e-06 2024-04-24T00:38:08.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886020+0000", 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6135000000000001e-05 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886027+0000", 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3379999999999999e-06 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886099+0000", 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1802000000000001e-05 2024-04-24T00:38:08.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886260+0000", 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016156800000000001 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.895327+0000", 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090669270000000007 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007496+0000", 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112168567 2024-04-24T00:38:08.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007520+0000", 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4646000000000001e-05 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007599+0000", 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8424999999999995e-05 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007668+0000", 2024-04-24T00:38:08.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9008000000000001e-05 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:157093 3.13 e197/106 ondisk, result = 0)", 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.171606646000001, 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090093925000000005, 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917416+0000", 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917426+0000", 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917465+0000", 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9295999999999998e-05 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917484+0000", 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:38:08.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007420+0000", 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089936099000000005 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007435+0000", 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4966999999999999e-05 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007507+0000", 2024-04-24T00:38:08.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2206000000000006e-05 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169765 3.13 3:c9e19d2e:::rbd_data.115ee24d4bb.0000000000000011:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.229241045999999, 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096474246999999999, 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.194 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169765 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859781+0000", 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859792+0000", 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859819+0000", 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7025999999999998e-05 2024-04-24T00:38:08.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859833+0000", 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3763e-05 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859956+0000", 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122989 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.860147+0000", 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190843 2024-04-24T00:38:08.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.866555+0000", 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064084540000000001 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.955923+0000", 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089367567999999994 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956056+0000", 2024-04-24T00:38:08.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013292600000000001 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956157+0000", 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101029 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956253+0000", 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5921999999999994e-05 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:173908 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [delete] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 18.420161774, 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97469691999999997, 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 173908 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668860+0000", 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7269999999999999e-06 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668868+0000", 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3799999999999994e-06 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668889+0000", 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0619999999999999e-05 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668906+0000", 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7303e-05 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669040+0000", 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134213 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669057+0000", 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7045999999999999e-05 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636000+0000", 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96694319299999998 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636222+0000", 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000221411 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636415+0000", 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193048 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.638630+0000", 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002215276 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.640380+0000", 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017497420000000001 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643297+0000", 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029172339999999999 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643383+0000", 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5566e-05 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643555+0000", 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017216099999999999 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.206 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:08.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:38:08.208 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:08.206+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:08.308 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:08.302+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:08.379 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:08.379 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:08.379 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:08.379 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:08.394 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:38:08.440 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:08.438+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:08.511 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:08.506+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:08.511 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:08.511 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:08.512 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:08.512 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:08.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:38:08.611 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:08.606+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:08.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:38:08.700 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:08.700 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:08.700 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.991128011, 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:38:08.701 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.702 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.703 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:38:08.704 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.705 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:08.706 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.99103008599999, 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.707 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.708 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.709 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.710 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.711 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:38:08.712 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 207.631650554, 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:08.713 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.714 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.715 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.716 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.717 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 184.22090642800001, 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.718 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.719 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.721 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.722 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:38:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.724 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.725 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:38:08.726 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 184.19563782200001, 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:08.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:38:08.730 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 170.342434102, 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:38:08.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:08.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 170.30982898799999, 2024-04-24T00:38:08.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.738 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:38:08.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.905254294, 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.740 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.741 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.742 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:38:08.743 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.744 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.84741406399999, 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.746 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:38:08.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 123.02710416399999, 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.750 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:38:08.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:38:08.755 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.997462111, 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:38:08.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:38:08.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.85094669999999, 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.760 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:38:08.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.766 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.062174142, 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:38:08.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 118.039030729, 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:38:08.772 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:38:08.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:08.776 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.165721981, 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:38:08.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 108.113039704, 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.782 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.783 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:166979 3.5 3:a3485662:::rbd_data.115e4ca9eff1.0000000000000010:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.713307751000002, 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.145524811, 2024-04-24T00:38:08.786 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 166979 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986190+0000", 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.323e-06 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986212+0000", 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1183999999999999e-05 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986246+0000", 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4409999999999998e-05 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986265+0000", 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986437+0000", 2024-04-24T00:38:08.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017130199999999999 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986581+0000", 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014409500000000001 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.997390+0000", 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010809456 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131432+0000", 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.134041997 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131521+0000", 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9024999999999996e-05 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131625+0000", 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103923 2024-04-24T00:38:08.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131713+0000", 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7672999999999999e-05 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:166979 3.5 e206/56 ondisk, result = 0)", 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.669772782999999, 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10200487, 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.792 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029726+0000", 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4899999999999999e-06 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029738+0000", 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2483000000000001e-05 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029784+0000", 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5920000000000001e-05 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029809+0000", 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5193000000000001e-05 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131491+0000", 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101681585 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131510+0000", 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9256999999999999e-05 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131728+0000", 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021794199999999999 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169782 3.d 3:b3c6180a:::rbd_data.115ee24d4bb.000000000000000f:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.566650242999998, 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.117826449, 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.796 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169782 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132848+0000", 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0770000000000002e-06 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132859+0000", 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132887+0000", 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8070000000000001e-05 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132908+0000", 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.09e-05 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133079+0000", 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017107900000000001 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133247+0000", 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168138 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.141250+0000", 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080027740000000007 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250223+0000", 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108972528 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250437+0000", 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021457900000000001 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250540+0000", 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102682 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250672+0000", 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013205899999999999 2024-04-24T00:38:08.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169782 3.d e210/106 ondisk, result = 0)", 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 46.537733910999997, 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088930206999999997, 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161764+0000", 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3259999999999998e-06 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161773+0000", 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5329999999999994e-06 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161803+0000", 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9867000000000001e-05 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161825+0000", 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2082999999999998e-05 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250323+0000", 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088498544999999998 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250342+0000", 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8983000000000001e-05 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250692+0000", 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034987000000000001 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.805 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:08.806 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:08.806 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:08.807 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:08.802+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:08.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:38:08.908 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:08.902+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:08.941 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:08.941 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:08.941 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:08.941 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:08.955 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:38:09.009 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:09.006+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:09.079 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:09.079 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:09.080 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:09.080 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:09.080 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:09.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:38:09.109 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:09.106+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:09.210 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:09.206+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 230.68473897199999, 2024-04-24T00:38:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:38:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 230.650701919, 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:38:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:38:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 198.53558792300001, 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:38:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:38:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:38:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:38:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 198.50813796200001, 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:38:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.58806612999999, 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 151.58179487199999, 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:38:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:38:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:38:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 78.841039107, 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 78.820147560999999, 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162938 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 16384~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434851653000003, 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070154141000000003, 2024-04-24T00:38:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162938 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0500000000000002e-07 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782146+0000", 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0489999999999997e-06 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782158+0000", 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1701e-05 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782169+0000", 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852078+0000", 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069909099000000002 2024-04-24T00:38:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852143+0000", 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4906999999999994e-05 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852296+0000", 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015313699999999999 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162939 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 49152~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434811556, 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070386700999999996, 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162939 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782186+0000", 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.924e-06 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782198+0000", 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2029e-05 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782208+0000", 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9240000000000002e-06 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852330+0000", 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070121292000000002 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852410+0000", 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9945999999999997e-05 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852568+0000", 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158763 2024-04-24T00:38:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162940 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434771566999999, 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070748154999999993, 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162940 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782226+0000", 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782239+0000", 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2516e-05 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782249+0000", 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064000000000001e-05 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852602+0000", 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070352985000000007 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852685+0000", 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3584999999999993e-05 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852970+0000", 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284412 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162941 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 114688~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434730733999999, 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071218216000000001, 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162941 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782263+0000", 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782267+0000", 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.172e-06 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782280+0000", 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2308e-05 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782290+0000", 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0674e-05 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853061+0000", 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070770437000000005 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853196+0000", 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013527100000000001 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853481+0000", 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028453700000000002 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162942 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 147456~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434689229999996, 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071662612000000001, 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162942 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782305+0000", 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3399999999999998e-07 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782308+0000", 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5480000000000002e-06 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782321+0000", 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2814e-05 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782331+0000", 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0363999999999999e-05 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853550+0000", 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071219062999999999 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853680+0000", 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129219 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853967+0000", 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028687 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162943 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 180224~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434648066000001, 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071922708000000002, 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162943 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782346+0000", 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6300000000000004e-07 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782350+0000", 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9129999999999996e-06 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782361+0000", 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1249e-05 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782370+0000", 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0829999999999993e-06 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854021+0000", 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071651144 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854119+0000", 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7157999999999994e-05 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854268+0000", 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014929799999999999 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162944 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 212992~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434610321000001, 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072204638000000002, 2024-04-24T00:38:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162944 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782384+0000", 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7800000000000001e-07 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782387+0000", 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7720000000000001e-06 2024-04-24T00:38:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782399+0000", 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1508000000000001e-05 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782409+0000", 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7550000000000006e-06 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854321+0000", 2024-04-24T00:38:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071912767000000002 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854437+0000", 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115282 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854587+0000", 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015087600000000001 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162945 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 245760~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434571536999997, 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072460953999999994, 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162945 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:38:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782427+0000", 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6210000000000003e-06 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782447+0000", 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0129999999999999e-05 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782457+0000", 2024-04-24T00:38:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.03e-05 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854619+0000", 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072161259000000005 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854739+0000", 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120542 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854883+0000", 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143354 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162946 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 278528~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434522553999997, 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072673115999999996, 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162946 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782493+0000", 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1645999999999999e-05 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782503+0000", 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4509999999999998e-06 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782511+0000", 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854939+0000", 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072427668000000001 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855054+0000", 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114957 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855144+0000", 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0175999999999999e-05 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162947 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 311296~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.297 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434472745000001, 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072916576999999996, 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162947 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782521+0000", 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782524+0000", 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.971e-06 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782532+0000", 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7940000000000003e-06 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782538+0000", 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855191+0000", 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072653180999999997 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855292+0000", 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100948 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855437+0000", 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014472700000000001 2024-04-24T00:38:09.301 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162948 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 56.434421534000002, 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073391995000000002, 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.302 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162948 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782573+0000", 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9700000000000005e-07 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782577+0000", 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782587+0000", 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5140000000000007e-06 2024-04-24T00:38:09.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782594+0000", 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.968e-06 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855774+0000", 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073180178999999998 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855862+0000", 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7793999999999993e-05 2024-04-24T00:38:09.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855964+0000", 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010183099999999999 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:175102 3.11 3:8ca8b224:::rbd_header.115ec7ed39de:head [watch reconnect cookie 140203704658000 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.5819323399999998, 2024-04-24T00:38:09.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98633510999999996, 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 175102 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635063+0000", 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.798e-06 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635070+0000", 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4719999999999998e-06 2024-04-24T00:38:09.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635088+0000", 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7853e-05 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635099+0000", 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1058e-05 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635160+0000", 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0868000000000001e-05 2024-04-24T00:38:09.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635172+0000", 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2459e-05 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.615851+0000", 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98067890000000002 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.616075+0000", 2024-04-24T00:38:09.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022393399999999999 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.621396+0000", 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005320768 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.311 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:09.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:38:09.313 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:09.310+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:09.413 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:09.410+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:09.485 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:09.485 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:09.485 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:09.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:38:09.540 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:09.538+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:09.610 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:38:09.617 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:09.610+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:09.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:38:09.671 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:09.672 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:09.672 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:09.672 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:09.672 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:09.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:38:09.736 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:09.734+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:09.770 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 6, 5, 0] out_osds: [7, 4] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:38:09.770 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:38:09.770 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-24T00:38:09.770 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.554299 2024-04-24T00:38:09.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0.5542987561991453 2024-04-24T00:38:09.799 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:09.800 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:09.815 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:09.815 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 268.88822365499999, 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:38:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:38:09.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:38:09.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:38:09.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 235.956257967, 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:38:09.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:38:09.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:38:09.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:38:09.828 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 235.955009219, 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.829 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:38:09.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.834 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 226.797385115, 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.835 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:38:09.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.840 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 226.78389057199999, 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:09.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 222.522109457, 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:38:09.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:38:09.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 213.26852331399999, 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:38:09.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:09.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 213.24084945000001, 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:38:09.857 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:38:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:09.862 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 208.74682662999999, 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:09.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:38:09.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 208.737817796, 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.868 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:38:09.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.23611884799999, 2024-04-24T00:38:09.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:09.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.876 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.236056502, 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.23554682299999, 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.881 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:38:09.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.23543576099999, 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.886 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:38:09.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:09.890 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.23537049500001, 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.894 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.23531042900001, 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:38:09.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.235253736, 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:38:09.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:38:09.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.23518440300001, 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:38:09.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:38:09.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.907 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.23509378399999, 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.908 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:09.912 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 189.235050371, 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:09.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:09.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:09.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.916 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:09.917 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:09.917 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:09.917 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:09.918 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:09.919 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:09.919 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:09.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:38:09.920 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:09.914+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:09.920 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:09.920 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:09.921 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:09.921 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:09.921 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:09.921 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:09.922 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:09.922 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:09.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:09.924 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:10.019 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:10.014+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:10.042 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:10.042 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:10.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:38:10.120 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:10.114+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:10.177 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:10.177 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:10.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:38:10.221 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:10.218+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:10.322 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:10.318+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:10.362 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:10.362 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:10.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:38:10.423 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:10.418+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:10.498 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:10.498 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:10.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:38:10.524 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:10.522+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:10.624 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:10.622+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:10.682 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:10.683 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:10.696 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.2 primary-affinity to 0.554299 (8363262) 2024-04-24T00:38:10.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:38:10.725 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:10.722+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:10.826 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:10.822+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:10.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:38:10.926 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:10.922+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:10.979 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:10.979 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:10.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:38:11.037 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:11.034+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:11.129 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:11.126+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:11.166 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:11.166 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:11.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:38:11.231 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:11.226+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:11.310 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:11.310 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:11.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:38:11.331 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:11.330+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:11.432 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:11.430+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:11.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:38:11.533 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:11.530+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:11.592 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:11.592 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:11.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:38:11.634 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:11.630+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:11.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:38:11.735 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:11.730+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:11.740 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:11.740 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:11.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:38:11.836 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:11.830+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:11.897 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:11.897 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:11.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:38:11.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:38:11.956 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:11.954+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:12.034 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:12.034 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:12.038 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:12.034+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:12.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:38:12.092 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:12.093 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:12.109 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:12.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:12.126 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:12.139 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:12.134+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:12.144 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:12.144 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:12.162 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:12.162 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:12.182 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:12.202 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:12.206 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:12.221 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:12.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10260 3.9 3:92a87828:::rbd_data.115e380d621c.0000000000000074:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 239.003792777, 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077537686999999994, 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10260 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.222 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214325+0000", 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214327+0000", 2024-04-24T00:38:12.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214339+0000", 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2217e-05 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214367+0000", 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7906e-05 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214382+0000", 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.438e-05 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214570+0000", 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000188622 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.214802+0000", 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023176800000000001 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.218544+0000", 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003742156 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291750+0000", 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073205835999999996 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291766+0000", 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5739999999999998e-05 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291814+0000", 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8192999999999998e-05 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.291863+0000", 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8532000000000002e-05 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.227 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 224.58200708300001, 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082331392000000003, 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636111+0000", 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636113+0000", 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.83e-06 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636127+0000", 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3263e-05 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636157+0000", 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0463e-05 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636171+0000", 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4017e-05 2024-04-24T00:38:12.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636298+0000", 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012713899999999999 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.636519+0000", 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022090099999999999 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.639801+0000", 2024-04-24T00:38:12.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032816569999999999 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.717554+0000", 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077752928999999998 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.718442+0000", 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088819300000000001 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:78086 3.e 3:772bfd2c:::rbd_data.115e805f80b3.0000000000000008:head [truncate 2883584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 163.43987951700001, 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78503430799999996, 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 78086 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778238+0000", 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778239+0000", 2024-04-24T00:38:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-04-24T00:38:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778244+0000", 2024-04-24T00:38:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2000000000000002e-06 2024-04-24T00:38:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778255+0000", 2024-04-24T00:38:12.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1049999999999999e-05 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778264+0000", 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7749999999999998e-06 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778417+0000", 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015280799999999999 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:12.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:28.778446+0000", 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9425999999999999e-05 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.560834+0000", 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.78238729900000004 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561049+0000", 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021519700000000001 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.561305+0000", 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025597499999999997 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.562275+0000", 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00097069699999999997 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563196+0000", 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092060099999999995 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563241+0000", 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4444999999999999e-05 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:29.563272+0000", 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1951e-05 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.241 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81501 3.e 3:749f5922:::rbd_data.115e6f07caf6.0000000000000001:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 158.30210773499999, 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11768512, 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81501 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916010+0000", 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.916012+0000", 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2170000000000001e-06 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920052+0000", 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040398730000000002 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920070+0000", 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7788e-05 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920081+0000", 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389000000000001e-05 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920154+0000", 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2645000000000007e-05 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920276+0000", 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012223600000000001 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.920389+0000", 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011314100000000001 2024-04-24T00:38:12.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.939151+0000", 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018761384999999998 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033590+0000", 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094439851000000005 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033652+0000", 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1358000000000004e-05 2024-04-24T00:38:12.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.033695+0000", 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3237e-05 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:81509 3.e 3:706aa345:::rbd_data.115e6f07caf6.0000000000000003:head [write 0~4194304 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 158.26462541699999, 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097481725000000005, 2024-04-24T00:38:12.247 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 81509 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953492+0000", 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.953494+0000", 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.872e-06 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959436+0000", 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059415329999999997 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959467+0000", 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1417000000000001e-05 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959483+0000", 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6137000000000002e-05 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959567+0000", 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4103000000000004e-05 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959741+0000", 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017390299999999999 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:33.959920+0000", 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178741 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.027210+0000", 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067289889000000006 2024-04-24T00:38:12.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050889+0000", 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0236787 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050940+0000", 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1545000000000002e-05 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:34.050974+0000", 2024-04-24T00:38:12.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3884999999999998e-05 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87727 3.11 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 154.60227955299999, 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14560906800000001, 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87727 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615838+0000", 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.615840+0000", 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9860000000000001e-06 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625625+0000", 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097851910000000004 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625652+0000", 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6968e-05 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625664+0000", 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1426999999999999e-05 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625766+0000", 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102188 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625981+0000", 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021510799999999999 2024-04-24T00:38:12.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.626314+0000", 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033316500000000001 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.749779+0000", 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.123465196 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761351+0000", 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011571141 2024-04-24T00:38:12.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761406+0000", 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5231e-05 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761447+0000", 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1467000000000001e-05 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87729 3.3 3:c641a3dd:::rbd_data.115e12b71652.0000000000000001:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 154.59242151699999, 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082837526999999994, 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87729 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625696+0000", 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.625698+0000", 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8059999999999999e-06 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645924+0000", 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.020226105000000001 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645968+0000", 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4305999999999999e-05 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.645991+0000", 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2405999999999999e-05 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646115+0000", 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012373000000000001 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646310+0000", 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000195021 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646425+0000", 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011583599999999999 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708407+0000", 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061982047999999998 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708426+0000", 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8440999999999998e-05 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708493+0000", 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6649999999999994e-05 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.708534+0000", 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1177999999999998e-05 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.264 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87732 3.11 3:8bfce594:::rbd_data.115e12b71652.0000000000000002:head [write 0~8388608 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 154.57207190899999, 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11595978899999999, 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87732 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646046+0000", 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646048+0000", 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5459999999999998e-06 2024-04-24T00:38:12.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659251+0000", 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013202579000000001 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659276+0000", 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5088000000000001e-05 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.659297+0000", 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1180000000000001e-05 2024-04-24T00:38:12.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694320+0000", 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035022360000000002 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694523+0000", 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00020301499999999999 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.694691+0000", 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016851 2024-04-24T00:38:12.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.750153+0000", 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055462111000000001 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761861+0000", 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707594999999999 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761946+0000", 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4767999999999999e-05 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.762006+0000", 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0037000000000001e-05 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:89922 3.19 e125/123 3:9e161a5e:::rbd_data.115e56aa02fc.0000000000000000:head v 125'316, mlcod=119'314)", 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:12.270 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.53802657400001, 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079627973000000005, 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680091+0000", 2024-04-24T00:38:12.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.680092+0000", 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4200000000000005e-07 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700013+0000", 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019921438999999999 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700102+0000", 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8592999999999997e-05 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700123+0000", 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1217e-05 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700295+0000", 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017225500000000001 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.700340+0000", 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4035000000000001e-05 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759656+0000", 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059316138999999997 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.759719+0000", 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3453000000000004e-05 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.274 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:89924 3.f 3:f211240d:::rbd_data.115e56aa02fc.0000000000000001:head [write 0~5242880 [fadvise_sequential+fadvise_nocache],zero 5767168~262144,zero 6291456~524288,zero 7077888~262144,zero 7864320~524288,zero 12845056~191410,write 13036466~21621 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.53571673799999, 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083022067000000005, 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 89924 2024-04-24T00:38:12.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682401+0000", 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.682404+0000", 2024-04-24T00:38:12.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690341+0000", 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079368770000000002 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690393+0000", 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2366e-05 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690419+0000", 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5933999999999999e-05 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690522+0000", 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102976 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.690653+0000", 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013136300000000001 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.691379+0000", 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00072622400000000003 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.750131+0000", 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058751286 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765276+0000", 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015145596000000001 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765361+0000", 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4790000000000006e-05 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:48.765423+0000", 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1934999999999996e-05 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:90274 3.1 3:81d935b6:::rbd_data.115e56aa02fc.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write 6647439~1859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-24T00:38:12.280 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 139.587998846, 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038603310000001, 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90274 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:12.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630119+0000", 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630120+0000", 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0890000000000001e-06 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630127+0000", 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1269999999999997e-06 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630140+0000", 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2503e-05 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630145+0000", 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8860000000000001e-06 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630261+0000", 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011510000000000001 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:52.630279+0000", 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8159999999999999e-05 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.627739+0000", 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99746076699999997 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628091+0000", 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035173399999999998 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.628528+0000", 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043722299999999998 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633464+0000", 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049351799999999999 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633655+0000", 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190929 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633840+0000", 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018539399999999999 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:53.633979+0000", 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013923900000000001 2024-04-24T00:38:12.286 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92307 3.13 3:cfec43d9:::rbd_data.115ee57427d8.0000000000000004:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.69860885, 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.151979112, 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.287 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92307 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519509+0000", 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519511+0000", 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.04e-06 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519523+0000", 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2053e-05 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519552+0000", 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8836999999999999e-05 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519573+0000", 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0746000000000001e-05 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519701+0000", 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012881899999999999 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.519835+0000", 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013337599999999999 2024-04-24T00:38:12.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.530736+0000", 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010901198000000001 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671197+0000", 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14046102899999999 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671320+0000", 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012258899999999999 2024-04-24T00:38:12.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671414+0000", 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4221999999999993e-05 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671488+0000", 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4202999999999994e-05 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92307 3.13 e141/106 ondisk, result = 0)", 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.655133053, 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10851801799999999, 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562985+0000", 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562987+0000", 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.43e-06 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.562997+0000", 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-04-24T00:38:12.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563032+0000", 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4678999999999998e-05 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.563056+0000", 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4117e-05 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671290+0000", 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10823387 2024-04-24T00:38:12.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671309+0000", 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9763e-05 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.671503+0000", 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193215 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.296 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94354 3.14 3:2b21f58b:::rbd_data.115eb3626b1c.0000000000000011:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 121.681159751, 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096070382999999995, 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94354 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536958+0000", 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536960+0000", 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536971+0000", 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1223000000000001e-05 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.536998+0000", 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7263e-05 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537014+0000", 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5193e-05 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537131+0000", 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117463 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.537287+0000", 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015557200000000001 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.543385+0000", 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0060981999999999998 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632694+0000", 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089308949999999998 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632821+0000", 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012668299999999999 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.632927+0000", 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106136 2024-04-24T00:38:12.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.633028+0000", 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101666 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106516 3.9 3:94c7fbe1:::rbd_data.115ec2448e7a.000000000000000d:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.70273029499999, 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13087674899999999, 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.302 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106516 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515387+0000", 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515390+0000", 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0999999999999998e-06 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515401+0000", 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1497e-05 2024-04-24T00:38:12.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515433+0000", 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1612999999999997e-05 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515447+0000", 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4423000000000001e-05 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515566+0000", 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118823 2024-04-24T00:38:12.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.515701+0000", 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013530200000000001 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.524755+0000", 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090539790000000002 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646163+0000", 2024-04-24T00:38:12.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12140764499999999 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646193+0000", 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0332999999999999e-05 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646226+0000", 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2883999999999999e-05 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646264+0000", 2024-04-24T00:38:12.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8149999999999999e-05 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106516 3.9 e153/21 ondisk, result = 0)", 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.66872787299999, 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096792074000000006, 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549390+0000", 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549392+0000", 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9700000000000002e-06 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549401+0000", 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549426+0000", 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5296e-05 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.549442+0000", 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6626999999999999e-05 2024-04-24T00:38:12.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646130+0000", 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096687476999999994 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646151+0000", 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1480000000000001e-05 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:19.646182+0000", 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0497e-05 2024-04-24T00:38:12.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:157093 3.13 3:cd0b7234:::rbd_data.115ee709dad1.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.332122636999998, 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12167272799999999, 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.312 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157093 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885995+0000", 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.885996+0000", 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.426e-06 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886003+0000", 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8859999999999998e-06 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886020+0000", 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6135000000000001e-05 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886027+0000", 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3379999999999999e-06 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886099+0000", 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1802000000000001e-05 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.886260+0000", 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016156800000000001 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.895327+0000", 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090669270000000007 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007496+0000", 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.112168567 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007520+0000", 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4646000000000001e-05 2024-04-24T00:38:12.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007599+0000", 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8424999999999995e-05 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007668+0000", 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9008000000000001e-05 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:157093 3.13 e197/106 ondisk, result = 0)", 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 65.300704598999999, 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090093925000000005, 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917413+0000", 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917416+0000", 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7199999999999998e-06 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917426+0000", 2024-04-24T00:38:12.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0227000000000001e-05 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917465+0000", 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9295999999999998e-05 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:06.917484+0000", 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8410000000000002e-05 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007420+0000", 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089936099000000005 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007435+0000", 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4966999999999999e-05 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:07.007507+0000", 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2206000000000006e-05 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169765 3.13 3:c9e19d2e:::rbd_data.115ee24d4bb.0000000000000011:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.358338998999997, 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096474246999999999, 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169765 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.322 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859779+0000", 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859781+0000", 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0870000000000001e-06 2024-04-24T00:38:12.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859792+0000", 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.164e-05 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859819+0000", 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7025999999999998e-05 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859833+0000", 2024-04-24T00:38:12.324 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3763e-05 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.859956+0000", 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122989 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.860147+0000", 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000190843 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.866555+0000", 2024-04-24T00:38:12.325 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064084540000000001 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.955923+0000", 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.089367567999999994 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956056+0000", 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013292600000000001 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956157+0000", 2024-04-24T00:38:12.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101029 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:21.956253+0000", 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5921999999999994e-05 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:173908 3.8 3:15129132:::rbd_data.115e6f07caf6.0000000000000002:head [delete] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 22.549259726999999, 2024-04-24T00:38:12.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97469691999999997, 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 173908 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668858+0000", 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668860+0000", 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7269999999999999e-06 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668868+0000", 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3799999999999994e-06 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668889+0000", 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0619999999999999e-05 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.668906+0000", 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7303e-05 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669040+0000", 2024-04-24T00:38:12.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134213 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:49.669057+0000", 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7045999999999999e-05 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636000+0000", 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96694319299999998 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636222+0000", 2024-04-24T00:38:12.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000221411 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.636415+0000", 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000193048 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.638630+0000", 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002215276 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.640380+0000", 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017497420000000001 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643297+0000", 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029172339999999999 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643383+0000", 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5566e-05 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:50.643555+0000", 2024-04-24T00:38:12.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017216099999999999 2024-04-24T00:38:12.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.334 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.334 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.334 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:12.335 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:12.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:38:12.337 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:12.337 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:12.330+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:12.436 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:12.434+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:12.465 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:12.465 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:12.465 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:12.465 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:12.479 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:38:12.568 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:12.566+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:12.638 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:12.634+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:12.645 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:12.645 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:12.646 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:12.646 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:12.646 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:12.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:38:12.713 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:38:12.738 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:12.734+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 225.121294519, 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:38:12.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:38:12.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.836 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 225.121196594, 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.837 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:38:12.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:38:12.843 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 211.76181706200001, 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:38:12.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:38:12.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 188.35107293600001, 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.850 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:38:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.855 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 188.32580433000001, 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:12.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.859 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.47260061, 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.860 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:38:12.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.865 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.43999549599999, 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:38:12.869 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 174.035420802, 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.870 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:38:12.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:38:12.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 173.97758057199999, 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:38:12.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:38:12.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.157270672, 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.880 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:38:12.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 127.12762861900001, 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.886 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:38:12.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.981113208, 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.890 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:12.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:12.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:12.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:38:12.896 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.19234065000001, 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:38:12.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:12.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:38:12.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 122.16919723700001, 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:38:12.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.905 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.906 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.29588848900001, 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.907 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:38:12.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.912 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 112.243206212, 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:12.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:38:12.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:166979 3.5 3:a3485662:::rbd_data.115e4ca9eff1.0000000000000010:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 54.843474258999997, 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.145524811, 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.917 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 166979 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:12.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986190+0000", 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.323e-06 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986212+0000", 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1183999999999999e-05 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986246+0000", 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4409999999999998e-05 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986265+0000", 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986437+0000", 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017130199999999999 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986581+0000", 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014409500000000001 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.997390+0000", 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010809456 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131432+0000", 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.134041997 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.921 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131521+0000", 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9024999999999996e-05 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131625+0000", 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103923 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131713+0000", 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7672999999999999e-05 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.922 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:166979 3.5 e206/56 ondisk, result = 0)", 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 54.799939291000001, 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10200487, 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029726+0000", 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4899999999999999e-06 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029738+0000", 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2483000000000001e-05 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029784+0000", 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5920000000000001e-05 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029809+0000", 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5193000000000001e-05 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131491+0000", 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101681585 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131510+0000", 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9256999999999999e-05 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131728+0000", 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021794199999999999 2024-04-24T00:38:12.926 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169782 3.d 3:b3c6180a:::rbd_data.115ee24d4bb.000000000000000f:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.696816751, 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.117826449, 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:12.927 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169782 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132848+0000", 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0770000000000002e-06 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132859+0000", 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132887+0000", 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8070000000000001e-05 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132908+0000", 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.09e-05 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133079+0000", 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017107900000000001 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133247+0000", 2024-04-24T00:38:12.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168138 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.141250+0000", 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080027740000000007 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250223+0000", 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108972528 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250437+0000", 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021457900000000001 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250540+0000", 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102682 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250672+0000", 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013205899999999999 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.932 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169782 3.d e210/106 ondisk, result = 0)", 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 50.667900418999999, 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088930206999999997, 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:12.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161764+0000", 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3259999999999998e-06 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.934 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161773+0000", 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5329999999999994e-06 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161803+0000", 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9867000000000001e-05 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161825+0000", 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2082999999999998e-05 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.935 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250323+0000", 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088498544999999998 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250342+0000", 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8983000000000001e-05 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250692+0000", 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034987000000000001 2024-04-24T00:38:12.936 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.937 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.937 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.937 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:12.937 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:12.937 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:12.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:38:12.939 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:12.934+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:13.039 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:13.034+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:13.069 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:13.070 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:13.070 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:13.070 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:13.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:38:13.172 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:13.170+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:13.240 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:13.234+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:13.249 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:13.249 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:13.249 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:13.249 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:13.249 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:13.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:38:13.341 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:13.338+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 234.90027010399999, 2024-04-24T00:38:13.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:38:13.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:13.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:13.441 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 234.86623305099999, 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.445 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 202.751119055, 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:38:13.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:38:13.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 202.723669094, 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:38:13.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:38:13.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.80359726200001, 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:38:13.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:38:13.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.79732600400001, 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:13.460 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:13.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:38:13.465 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 83.056570238999996, 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.466 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:38:13.469 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:38:13.470 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.471 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 83.035678692999994, 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:38:13.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:38:13.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162938 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 16384~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650382784999998, 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070154141000000003, 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162938 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.476 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:13.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0500000000000002e-07 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782146+0000", 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0489999999999997e-06 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782158+0000", 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1701e-05 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782169+0000", 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852078+0000", 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069909099000000002 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852143+0000", 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4906999999999994e-05 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852296+0000", 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015313699999999999 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162939 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 49152~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650342688000002, 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070386700999999996, 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.480 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162939 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:13.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782186+0000", 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.924e-06 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782198+0000", 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2029e-05 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782208+0000", 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9240000000000002e-06 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852330+0000", 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070121292000000002 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852410+0000", 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9945999999999997e-05 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852568+0000", 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158763 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162940 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:13.484 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650302699000001, 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070748154999999993, 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162940 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:13.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782226+0000", 2024-04-24T00:38:13.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782239+0000", 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2516e-05 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782249+0000", 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064000000000001e-05 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852602+0000", 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070352985000000007 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852685+0000", 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3584999999999993e-05 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852970+0000", 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284412 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162941 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 114688~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650261866000001, 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071218216000000001, 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162941 2024-04-24T00:38:13.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782263+0000", 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782267+0000", 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.172e-06 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782280+0000", 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2308e-05 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782290+0000", 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0674e-05 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853061+0000", 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070770437000000005 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853196+0000", 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013527100000000001 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853481+0000", 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028453700000000002 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162942 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 147456~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650220361999999, 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071662612000000001, 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.493 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162942 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.494 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782305+0000", 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3399999999999998e-07 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782308+0000", 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5480000000000002e-06 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782321+0000", 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2814e-05 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782331+0000", 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0363999999999999e-05 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853550+0000", 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071219062999999999 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853680+0000", 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129219 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853967+0000", 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028687 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.497 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162943 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 180224~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650179198000004, 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071922708000000002, 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162943 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782346+0000", 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6300000000000004e-07 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782350+0000", 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9129999999999996e-06 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782361+0000", 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1249e-05 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782370+0000", 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0829999999999993e-06 2024-04-24T00:38:13.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854021+0000", 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071651144 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854119+0000", 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7157999999999994e-05 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854268+0000", 2024-04-24T00:38:13.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014929799999999999 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162944 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 212992~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650141453000003, 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072204638000000002, 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.502 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162944 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:13.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782384+0000", 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7800000000000001e-07 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782387+0000", 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7720000000000001e-06 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.504 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782399+0000", 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1508000000000001e-05 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782409+0000", 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7550000000000006e-06 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854321+0000", 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071912767000000002 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.505 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854437+0000", 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115282 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854587+0000", 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015087600000000001 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162945 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 245760~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.506 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650102668999999, 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072460953999999994, 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162945 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:13.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782427+0000", 2024-04-24T00:38:13.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6210000000000003e-06 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782447+0000", 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0129999999999999e-05 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782457+0000", 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.03e-05 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854619+0000", 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072161259000000005 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854739+0000", 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120542 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854883+0000", 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143354 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162946 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 278528~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650053686, 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072673115999999996, 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162946 2024-04-24T00:38:13.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782493+0000", 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1645999999999999e-05 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782503+0000", 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4509999999999998e-06 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782511+0000", 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854939+0000", 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072427668000000001 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855054+0000", 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114957 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855144+0000", 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0175999999999999e-05 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162947 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 311296~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.650003877000003, 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072916576999999996, 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.515 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162947 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782521+0000", 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782524+0000", 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.971e-06 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782532+0000", 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7940000000000003e-06 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782538+0000", 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855191+0000", 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072653180999999997 2024-04-24T00:38:13.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855292+0000", 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100948 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855437+0000", 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014472700000000001 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.519 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162948 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 60.649952665999997, 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073391995000000002, 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162948 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782573+0000", 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9700000000000005e-07 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782577+0000", 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782587+0000", 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5140000000000007e-06 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782594+0000", 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.968e-06 2024-04-24T00:38:13.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855774+0000", 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073180178999999998 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855862+0000", 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7793999999999993e-05 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855964+0000", 2024-04-24T00:38:13.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010183099999999999 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:175102 3.11 3:8ca8b224:::rbd_header.115ec7ed39de:head [watch reconnect cookie 140203704658000 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 9.7974634720000005, 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98633510999999996, 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:13.524 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 175102 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635063+0000", 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.798e-06 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635070+0000", 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4719999999999998e-06 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635088+0000", 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7853e-05 2024-04-24T00:38:13.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635099+0000", 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1058e-05 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635160+0000", 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0868000000000001e-05 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635172+0000", 2024-04-24T00:38:13.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2459e-05 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.615851+0000", 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98067890000000002 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.616075+0000", 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022393399999999999 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:13.528 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.621396+0000", 2024-04-24T00:38:13.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005320768 2024-04-24T00:38:13.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.529 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:13.529 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:13.529 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:13.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:38:13.531 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:13.526+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:13.632 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:13.626+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:13.658 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:13.659 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:13.659 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:13.659 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:13.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:38:13.764 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:13.762+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:13.833 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:13.830+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:13.837 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:13.837 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:13.837 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:13.838 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:13.838 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:13.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:38:13.934 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:13.930+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:14.031 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:14.031 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:14.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:14.031 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:14.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.031 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:38:14.031 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:14.031 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 273.10029419799997, 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:14.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:38:14.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.036 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.037 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 240.16832851000001, 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.038 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:38:14.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.043 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 240.16707976199999, 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:38:14.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:38:14.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:14.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 231.00945565800001, 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:38:14.049 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:38:14.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:38:14.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:38:14.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:38:14.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 230.995961115, 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:38:14.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:38:14.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 226.73418000000001, 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:38:14.059 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:38:14.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:38:14.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:38:14.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 217.480593857, 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.065 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.070 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 217.45291999299999, 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.075 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 212.958897173, 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:38:14.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:38:14.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:14.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:14.081 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 212.94988833900001, 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:38:14.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.085 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.448189391, 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.44812704500001, 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:38:14.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.447617366, 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.094 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:38:14.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.447506304, 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:38:14.098 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.102 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.44744103799999, 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:14.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.106 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.44738097199999, 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.107 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:38:14.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:38:14.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:38:14.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.44732427899999, 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:14.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:38:14.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.44725494599999, 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:38:14.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.447164327, 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:38:14.120 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.124 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 193.44712091400001, 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:14.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.128 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:14.129 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:14.129 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:14.130 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:38:14.131 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:14.126+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:14.232 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:14.230+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:14.266 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:14.266 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:38:14.266 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:38:14.266 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:14.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:38:14.333 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:14.330+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:14.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:38:14.406 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:14.407 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:38:14.407 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:38:14.407 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:38:14.407 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:14.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:38:14.434 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:14.430+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:14.478 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:14.479 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:14.495 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:14.496 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:14.512 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:14.513 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:14.530 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:14.530 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:14.535 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:14.530+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:14.551 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:14.551 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:14.551 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:14.552 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:14.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:38:14.572 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:14.573 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:14.594 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:14.594 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:14.624 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:14.624 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:14.636 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:14.634+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:14.695 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:14.696 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:14.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:38:14.737 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:14.734+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:14.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T00:38:14.837 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:14.834+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:14.875 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:14.875 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:14.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:38:14.938 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:14.934+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:15.015 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:15.015 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:15.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:38:15.073 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:15.070+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:15.140 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:15.138+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:15.141 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:15.142 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:15.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:38:15.240 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:15.238+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:15.321 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:15.321 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:15.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:38:15.381 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:15.378+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:15.442 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:15.438+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:15.459 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:15.459 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:15.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:38:15.542 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:15.538+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:15.643 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:15.638+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:15.643 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:15.644 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:15.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:38:15.726 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 6, 5, 0] out_osds: [7, 4] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-04-24T00:38:15.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T00:38:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-04-24T00:38:15.744 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:15.738+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:15.838 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:15.838 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:15.845 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:15.842+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:15.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:38:15.945 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:15.942+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:16.011 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:16.012 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:16.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:38:16.072 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:16.070+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:16.147 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:16.142+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:16.154 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:16.154 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:16.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:38:16.248 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:16.242+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:16.334 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:16.335 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:16.349 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:38:16.349 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:16.346+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:16.450 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:16.446+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:16.473 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:16.473 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:16.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:38:16.576 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:16.574+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:16.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T00:38:16.652 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:16.650+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:16.656 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:16.656 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:16.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:38:16.742 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T00:38:16.756 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:16.754+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:16.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T00:38:16.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:16.817 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:16.835 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:16.836 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:16.838 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:16.839 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:16.854 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:16.850+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:16.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:38:16.857 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:16.874 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:16.874 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:16.892 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:16.892 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:16.913 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:16.913 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:16.932 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:16.932 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:16.949 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:16.949 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:16.954 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:16.950+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:17.030 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21646 3.1d 3:bbdb68ad:::rbd_data.115eeb9272a5.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:38:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:17.030 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 229.31906770099999, 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0287072850000001, 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21646 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:17.031 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708368+0000", 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708370+0000", 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5739999999999998e-06 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.032 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708380+0000", 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9219999999999992e-06 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708402+0000", 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2025999999999999e-05 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708417+0000", 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4936999999999999e-05 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.033 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714063+0000", 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005645181 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714076+0000", 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2979999999999999e-05 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.727570+0000", 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013494203 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.034 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728022+0000", 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000452576 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736897+0000", 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088747319999999998 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.737075+0000", 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178154 2024-04-24T00:38:17.035 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21647 3.1d 3:b92b5540:::rbd_header.115eeb9272a5:head [watch reconnect cookie 140203906033248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 229.31896977599999, 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031319367, 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.036 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21647 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708466+0000", 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.037 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708467+0000", 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0720000000000001e-06 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708472+0000", 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6019999999999999e-06 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708487+0000", 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4436e-05 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.038 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.708500+0000", 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2634e-05 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714111+0000", 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056116930000000001 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.714120+0000", 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2700000000000004e-06 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.039 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728114+0000", 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.013994689 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.728450+0000", 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00033623700000000003 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736512+0000", 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080620259999999999 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.040 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.736711+0000", 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019862499999999999 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738245+0000", 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015340830000000001 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.738332+0000", 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6520000000000005e-05 2024-04-24T00:38:17.041 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:28.739785+0000", 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00145348 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30133 3.1f 3:f9e089b6:::rbd_data.115e8210e51a.0000000000000037:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 215.959590244, 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073195128999999998, 2024-04-24T00:38:17.042 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30133 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.043 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067845+0000", 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067846+0000", 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6199999999999996e-07 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067852+0000", 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.48e-06 2024-04-24T00:38:17.044 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067873+0000", 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0990999999999999e-05 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067883+0000", 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0587999999999999e-05 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067987+0000", 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104147 2024-04-24T00:38:17.045 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068165+0000", 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017732699999999999 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071691+0000", 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035258749999999999 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.046 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140893+0000", 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069202765999999999 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140917+0000", 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3261999999999999e-05 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.140980+0000", 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3388999999999999e-05 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.047 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.141040+0000", 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0442000000000002e-05 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:65822 3.5 3:a79d13d7:::rbd_data.115e8561432.0000000000000103:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 192.548846118, 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10150743499999999, 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.048 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 65822 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478589+0000", 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478592+0000", 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478605+0000", 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2622999999999999e-05 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478636+0000", 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1420000000000001e-05 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478652+0000", 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.59e-05 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478776+0000", 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012347500000000001 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.478921+0000", 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014519500000000001 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.485256+0000", 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0063352649999999996 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579895+0000", 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094638605000000001 2024-04-24T00:38:17.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579926+0000", 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1127000000000002e-05 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580020+0000", 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4429e-05 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.580097+0000", 2024-04-24T00:38:17.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6705999999999994e-05 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:65822 3.5 e85/56 ondisk, result = 0)", 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 192.523577512, 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076052461000000002, 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503858+0000", 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503860+0000", 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.154e-06 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503868+0000", 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1259999999999998e-06 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503900+0000", 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2107000000000003e-05 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.503916+0000", 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:38:17.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579838+0000", 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.075922445000000005 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579855+0000", 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7275999999999999e-05 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:04.579911+0000", 2024-04-24T00:38:17.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5229999999999998e-05 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72443 3.1f 3:fea97212:::rbd_data.115efb339e04.0000000000000003:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.67037379199999, 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.116826912, 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.058 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72443 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357062+0000", 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357064+0000", 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.776e-06 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357073+0000", 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8940000000000005e-06 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357101+0000", 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7296000000000001e-05 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357114+0000", 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3617e-05 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357246+0000", 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000131872 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.357512+0000", 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000265262 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.365771+0000", 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082592889999999995 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473676+0000", 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107905588 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473710+0000", 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3405999999999998e-05 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473804+0000", 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4555999999999994e-05 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473889+0000", 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4356000000000003e-05 2024-04-24T00:38:17.063 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72443 3.1f e98/59 ondisk, result = 0)", 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.63776867799999, 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.084027332999999996, 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389667+0000", 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389669+0000", 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389677+0000", 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6400000000000003e-06 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389707+0000", 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.917e-05 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.389722+0000", 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5926999999999998e-05 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473646+0000", 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.083923161999999996 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473665+0000", 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9613999999999999e-05 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.473694+0000", 2024-04-24T00:38:17.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8963e-05 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:72566 3.1b 3:d9c52512:::rbd_data.115efb339e04.0000000000000052:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.233193984, 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.172824054, 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.068 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 72566 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794242+0000", 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794243+0000", 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7400000000000001e-06 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794253+0000", 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0139999999999999e-05 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794279+0000", 2024-04-24T00:38:17.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5403000000000001e-05 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794293+0000", 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.418e-05 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794475+0000", 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018188199999999999 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.794735+0000", 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000259798 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.804445+0000", 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097103880000000004 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966942+0000", 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162496693 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966985+0000", 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3282999999999999e-05 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967026+0000", 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0624999999999998e-05 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967066+0000", 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9922000000000002e-05 2024-04-24T00:38:17.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:72566 3.1b e98/53 ondisk, result = 0)", 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 178.17535375400001, 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11498936999999999, 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852082+0000", 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852084+0000", 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.294e-06 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.075 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852093+0000", 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3829999999999992e-06 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852125+0000", 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2143e-05 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.852141+0000", 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6252999999999998e-05 2024-04-24T00:38:17.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966973+0000", 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114832586 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.966980+0000", 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9430000000000003e-06 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:18.967071+0000", 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0767999999999996e-05 2024-04-24T00:38:17.077 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92308 3.d 3:b501a5ab:::rbd_data.115ee57427d8.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 131.355043854, 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12892094500000001, 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.078 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92308 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672392+0000", 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672395+0000", 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6799999999999999e-06 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672411+0000", 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6033e-05 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672447+0000", 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5472999999999997e-05 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672464+0000", 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7397e-05 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672660+0000", 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019520099999999999 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.672925+0000", 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00026575500000000001 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.681625+0000", 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086997250000000002 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800831+0000", 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11920584200000001 2024-04-24T00:38:17.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801090+0000", 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00025897000000000002 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801183+0000", 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3542999999999995e-05 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.801313+0000", 2024-04-24T00:38:17.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129326 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:92308 3.d e141/106 ondisk, result = 0)", 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 131.325401801, 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098802147000000007, 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702034+0000", 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702035+0000", 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3200000000000001e-06 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702042+0000", 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.956e-06 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702064+0000", 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2308000000000001e-05 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.702079+0000", 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4647000000000001e-05 2024-04-24T00:38:17.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800818+0000", 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098738813999999994 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800826+0000", 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8900000000000007e-06 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.800836+0000", 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:92330 3.1 3:816d46d7:::rbd_data.115ee57427d8.0000000000000002:head [truncate 9437184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 131.17888639, 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87569376099999996, 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.088 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 92330 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848549+0000", 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848551+0000", 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.466e-06 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848557+0000", 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.55e-06 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848580+0000", 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2504e-05 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848595+0000", 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5610000000000001e-05 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848784+0000", 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018913799999999999 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:17.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:05.848814+0000", 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9283000000000001e-05 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721033+0000", 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87221910400000002 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721488+0000", 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045561000000000001 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.721926+0000", 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043771599999999999 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723924+0000", 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001997778 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.723977+0000", 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3417e-05 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724132+0000", 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154506 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:06.724243+0000", 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111079 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.094 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:94356 3.7 3:e14388a3:::rbd_data.115eb3626b1c.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.390113832, 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096704107999999997, 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 94356 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637322+0000", 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637324+0000", 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637334+0000", 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0633e-05 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637363+0000", 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8816e-05 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637378+0000", 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4347e-05 2024-04-24T00:38:17.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637491+0000", 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113676 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.637629+0000", 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013812500000000001 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.643438+0000", 2024-04-24T00:38:17.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058089029999999998 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733633+0000", 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090194337999999999 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733835+0000", 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000202065 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733944+0000", 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109549 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734026+0000", 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699000000000005e-05 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:94356 3.7 e146/106 ondisk, result = 0)", 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.366970419, 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073576381999999996, 2024-04-24T00:38:17.100 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660465+0000", 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660467+0000", 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8899999999999999e-06 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660475+0000", 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9030000000000004e-06 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660506+0000", 2024-04-24T00:38:17.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1010000000000003e-05 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.660520+0000", 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4343e-05 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733731+0000", 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073210681999999999 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.733747+0000", 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5662999999999999e-05 2024-04-24T00:38:17.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:10.734042+0000", 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029489099999999998 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:106849 3.1f 3:fba906fa:::rbd_data.115ec2448e7a.0000000000000000:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 116.493661671, 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17600507500000001, 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.104 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106849 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533774+0000", 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533776+0000", 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3369999999999998e-06 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533788+0000", 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2243e-05 2024-04-24T00:38:17.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533820+0000", 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1436000000000002e-05 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533835+0000", 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.52e-05 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.533969+0000", 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134015 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.534121+0000", 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000152313 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2", 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.546970+0000", 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012848177000000001 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709575+0000", 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.162605371 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709606+0000", 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0898999999999997e-05 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709695+0000", 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8670000000000003e-05 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709779+0000", 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4413999999999995e-05 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:106849 3.1f e154/59 ondisk, result = 0)", 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 116.440979394, 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12313595500000001, 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.110 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586456+0000", 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586458+0000", 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3099999999999999e-06 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586468+0000", 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6539999999999994e-06 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586505+0000", 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6683000000000001e-05 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.586530+0000", 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5236e-05 2024-04-24T00:38:17.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709550+0000", 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12302019 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709564+0000", 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3947999999999999e-05 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:20.709592+0000", 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7934000000000001e-05 2024-04-24T00:38:17.113 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:166979 3.5 3:a3485662:::rbd_data.115e4ca9eff1.0000000000000010:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 59.041247441000003, 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.145524811, 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.114 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 166979 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986188+0000", 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986190+0000", 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.323e-06 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986212+0000", 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1183999999999999e-05 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986246+0000", 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4409999999999998e-05 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.116 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986265+0000", 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9423e-05 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986437+0000", 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017130199999999999 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.986581+0000", 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014409500000000001 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:17.997390+0000", 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010809456 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131432+0000", 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.134041997 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131521+0000", 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9024999999999996e-05 2024-04-24T00:38:17.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131625+0000", 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000103923 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131713+0000", 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7672999999999999e-05 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:166979 3.5 e206/56 ondisk, result = 0)", 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 58.997712473, 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10200487, 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029723+0000", 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029726+0000", 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4899999999999999e-06 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029738+0000", 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2483000000000001e-05 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029784+0000", 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5920000000000001e-05 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.029809+0000", 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5193000000000001e-05 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131491+0000", 2024-04-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101681585 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131510+0000", 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9256999999999999e-05 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:18.131728+0000", 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021794199999999999 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:169782 3.d 3:b3c6180a:::rbd_data.115ee24d4bb.000000000000000f:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 54.894589932999999, 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.117826449, 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169782 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132846+0000", 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132848+0000", 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0770000000000002e-06 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132859+0000", 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1562999999999999e-05 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132887+0000", 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8070000000000001e-05 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.132908+0000", 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.09e-05 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133079+0000", 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017107900000000001 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.133247+0000", 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168138 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 6", 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.141250+0000", 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0080027740000000007 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250223+0000", 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108972528 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250437+0000", 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021457900000000001 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250540+0000", 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000102682 2024-04-24T00:38:17.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250672+0000", 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013205899999999999 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:169782 3.d e210/106 ondisk, result = 0)", 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 54.865673600999997, 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088930206999999997, 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.129 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161762+0000", 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161764+0000", 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3259999999999998e-06 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161773+0000", 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5329999999999994e-06 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161803+0000", 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9867000000000001e-05 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.161825+0000", 2024-04-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2082999999999998e-05 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250323+0000", 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.088498544999999998 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250342+0000", 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8983000000000001e-05 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:22.250692+0000", 2024-04-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034987000000000001 2024-04-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:17.134 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:38:17.135 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:17.130+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:17.235 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:17.234+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:17.259 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:17.259 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:17.259 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:17.260 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:17.273 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:38:17.336 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:17.334+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:17.390 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:17.390 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:17.390 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:17.390 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:17.390 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:17.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:38:17.448 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-04-24T00:38:17.446+0000 7f185814e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25216) UID: 0 2024-04-24T00:38:17.538 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-04-24T00:38:17.534+0000 7f63e418c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25168) UID: 0 2024-04-24T00:38:17.574 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:17.574 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:16468 3.1e 3:790b0434:::rbd_data.115e8003c269.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 239.03958002900001, 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125006482, 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.575 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 16468 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532254+0000", 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532257+0000", 2024-04-24T00:38:17.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4430000000000002e-06 2024-04-24T00:38:17.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532269+0000", 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2266000000000001e-05 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532301+0000", 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1912999999999998e-05 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532323+0000", 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2237000000000001e-05 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532440+0000", 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000117419 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.532581+0000", 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014032199999999999 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.541168+0000", 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085872569999999992 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657095+0000", 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11592709499999999 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657117+0000", 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657180+0000", 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3262000000000005e-05 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657261+0000", 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0821000000000004e-05 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.581 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:16468 3.1e e45/44 ondisk, result = 0)", 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 239.00554297599999, 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090814609000000004, 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566291+0000", 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566294+0000", 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5000000000000002e-06 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566304+0000", 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8900000000000002e-06 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566337+0000", 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3346000000000002e-05 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.566357+0000", 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9899999999999999e-05 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657074+0000", 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.090717583000000004 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657087+0000", 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2663999999999999e-05 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:18.657106+0000", 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8726e-05 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.585 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:35620 3.10 3:087bafcb:::rbd_data.115ec5833ea9.0000000000000055:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 206.89042898, 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14291147000000001, 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 35620 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681405+0000", 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681407+0000", 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409e-06 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681413+0000", 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8460000000000002e-06 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681430+0000", 2024-04-24T00:38:17.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6265000000000001e-05 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681442+0000", 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1776e-05 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681512+0000", 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0687999999999998e-05 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.681583+0000", 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0821999999999994e-05 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.689479+0000", 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078960929999999999 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824137+0000", 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13465772400000001 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824208+0000", 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1038999999999995e-05 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824264+0000", 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5843e-05 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824317+0000", 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2964999999999999e-05 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.591 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:35620 3.10 e74/59 ondisk, result = 0)", 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 206.86297901899999, 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115291912, 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708855+0000", 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708858+0000", 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3659999999999999e-06 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708866+0000", 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6770000000000001e-06 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708893+0000", 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6642999999999999e-05 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.708909+0000", 2024-04-24T00:38:17.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5868000000000001e-05 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824111+0000", 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.115202718 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824123+0000", 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1221e-05 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:50.824147+0000", 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4419000000000001e-05 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.4446.0:87727 3.11 e115/106 3:8cbc02f6:::rbd_data.115e12b71652.0000000000000000:head v 115'358, mlcod=113'356)", 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 159.942907187, 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132224482, 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.596 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.628927+0000", 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.92e-07 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635067+0000", 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061394010000000001 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635093+0000", 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6743000000000001e-05 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635107+0000", 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635221+0000", 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011396999999999999 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635259+0000", 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7771000000000001e-05 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761099+0000", 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.125840321 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.761152+0000", 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2509999999999997e-05 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:87736 3.a 3:54a649ca:::rbd_data.115e12b71652.0000000000000003:head [write 0~7723400 [fadvise_sequential+fadvise_nocache],zero 8388608~98304,zero 8519680~147456,zero 8699904~147456,zero 8880128~147456,zero 9060352~7628,write 9067980~1433 [fadvise_sequential+fadvise_nocache],zero 9069413~138395,zero 9240576~147456,zero 9420800~147456,zero 9601024~147456,zero 9781248~147456,zero 9961472~147456,zero 10141696~147456,zero 10321920~147456,zero 10502144~147456,zero 10682368~147456,zero 10862592~147456,zero 11042816~147456,zero 11223040~147456,zero 11403264~147456,zero 11583488~147456,zero 11763712~147456,zero 11943936~147456,zero 12124160~147456,zero 12304384~104496,write 12408880~4080 [fadvise_sequential+fadvise_nocache],zero 12412960~38880,zero 12484608~147456,zero 12664832~147456,zero 12845056~147456,zero 13025280~147456,zero 13205504~147456,zero 13385728~147456,zero 13565952~147456,zero 13746176~147456,zero 13926400~147456,zero 14106624~147456,zero 14286848~147456,zero 14467072~147456,zero 14647296~147456,zero 14827520~147456,zero 15007744~147456,zero 15187968~147456,zero 15368192~147456,zero 15548416~147456,zero 15728640~4639,write 15733279~58442 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-24T00:38:17.600 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 159.936635929, 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096582640999999997, 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 87736 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635198+0000", 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.635201+0000", 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4210000000000002e-06 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646120+0000", 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010918898999999999 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646145+0000", 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5728000000000002e-05 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646156+0000", 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1086e-05 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646273+0000", 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011686 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.646623+0000", 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034933499999999999 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.648738+0000", 2024-04-24T00:38:17.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021156289999999999 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731388+0000", 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.082649981999999997 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731612+0000", 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022362200000000001 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731724+0000", 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111962 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:37.731781+0000", 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7117000000000002e-05 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:143836 3.1e 3:79121558:::rbd_data.115e8dbf22aa.000000000000004c:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:38:17.606 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 87.195880164000002, 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101153164, 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 143836 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375954+0000", 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375956+0000", 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.373e-06 2024-04-24T00:38:17.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375968+0000", 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1347e-05 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.375997+0000", 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9329999999999999e-05 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376014+0000", 2024-04-24T00:38:17.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6858999999999999e-05 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376112+0000", 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8356000000000005e-05 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.376287+0000", 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174298 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.380956+0000", 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046691809999999997 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476892+0000", 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.095936455000000004 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476926+0000", 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3420000000000002e-05 2024-04-24T00:38:17.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477026+0000", 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010053899999999999 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.477107+0000", 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1006000000000003e-05 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:143836 3.1e e183/44 ondisk, result = 0)", 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 87.174988618, 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080063013000000002, 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396846+0000", 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396848+0000", 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9929999999999998e-06 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396856+0000", 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8640000000000002e-06 2024-04-24T00:38:17.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396891+0000", 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4143000000000002e-05 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.396908+0000", 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7518999999999999e-05 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476861+0000", 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.079953110999999993 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476879+0000", 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8104000000000001e-05 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:36:50.476909+0000", 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9278999999999999e-05 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.616 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162938 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 16384~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789692709999997, 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070154141000000003, 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.617 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162938 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782142+0000", 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0500000000000002e-07 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782146+0000", 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0489999999999997e-06 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782158+0000", 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1701e-05 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782169+0000", 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852078+0000", 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069909099000000002 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852143+0000", 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4906999999999994e-05 2024-04-24T00:38:17.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852296+0000", 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015313699999999999 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162939 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 49152~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789652613000001, 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070386700999999996, 2024-04-24T00:38:17.621 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162939 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782182+0000", 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782186+0000", 2024-04-24T00:38:17.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.924e-06 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782198+0000", 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2029e-05 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782208+0000", 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9240000000000002e-06 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852330+0000", 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070121292000000002 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852410+0000", 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9945999999999997e-05 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852568+0000", 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000158763 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.625 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162940 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 81920~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789612624, 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070748154999999993, 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162940 2024-04-24T00:38:17.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782222+0000", 2024-04-24T00:38:17.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782226+0000", 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7759999999999999e-06 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782239+0000", 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2516e-05 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782249+0000", 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0064000000000001e-05 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852602+0000", 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070352985000000007 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852685+0000", 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3584999999999993e-05 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.852970+0000", 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284412 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162941 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 114688~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789571791, 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071218216000000001, 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.630 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162941 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782262+0000", 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782263+0000", 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782267+0000", 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.172e-06 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782280+0000", 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2308e-05 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782290+0000", 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0674e-05 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853061+0000", 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.070770437000000005 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853196+0000", 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013527100000000001 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853481+0000", 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028453700000000002 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162942 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 147456~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.634 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789530287000005, 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071662612000000001, 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162942 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782304+0000", 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782305+0000", 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3399999999999998e-07 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782308+0000", 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5480000000000002e-06 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782321+0000", 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2814e-05 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782331+0000", 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0363999999999999e-05 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853550+0000", 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071219062999999999 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853680+0000", 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129219 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.853967+0000", 2024-04-24T00:38:17.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028687 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162943 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 180224~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789489122999996, 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071922708000000002, 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162943 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782345+0000", 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782346+0000", 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6300000000000004e-07 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782350+0000", 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9129999999999996e-06 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782361+0000", 2024-04-24T00:38:17.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1249e-05 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782370+0000", 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0829999999999993e-06 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854021+0000", 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071651144 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854119+0000", 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7157999999999994e-05 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854268+0000", 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014929799999999999 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162944 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 212992~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789451377999995, 2024-04-24T00:38:17.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072204638000000002, 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162944 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:17.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782383+0000", 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782384+0000", 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7800000000000001e-07 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782387+0000", 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7720000000000001e-06 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782399+0000", 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1508000000000001e-05 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782409+0000", 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7550000000000006e-06 2024-04-24T00:38:17.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854321+0000", 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.071912767000000002 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854437+0000", 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115282 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854587+0000", 2024-04-24T00:38:17.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015087600000000001 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162945 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 245760~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789412593999998, 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072460953999999994, 2024-04-24T00:38:17.648 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162945 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782422+0000", 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4799999999999997e-07 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782427+0000", 2024-04-24T00:38:17.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6210000000000003e-06 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782447+0000", 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0129999999999999e-05 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782457+0000", 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.03e-05 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854619+0000", 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072161259000000005 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854739+0000", 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120542 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854883+0000", 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143354 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.652 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162946 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 278528~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789363610999999, 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072673115999999996, 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.653 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162946 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782471+0000", 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6499999999999998e-07 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782493+0000", 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1645999999999999e-05 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782503+0000", 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4509999999999998e-06 2024-04-24T00:38:17.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782511+0000", 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3529999999999995e-06 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.854939+0000", 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072427668000000001 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855054+0000", 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114957 2024-04-24T00:38:17.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855144+0000", 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0175999999999999e-05 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162947 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 311296~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789313801999995, 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072916576999999996, 2024-04-24T00:38:17.657 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162947 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782520+0000", 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782521+0000", 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7599999999999997e-07 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782524+0000", 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.971e-06 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782532+0000", 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7940000000000003e-06 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782538+0000", 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855191+0000", 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.072653180999999997 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855292+0000", 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100948 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855437+0000", 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014472700000000001 2024-04-24T00:38:17.661 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:162948 3.a 3:5305f41a:::rbd_data.115e135fe8e7.0000000000000026:50 [read 344064~16384 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e202)", 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 64.789262590999996, 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073391995000000002, 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.662 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 162948 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782572+0000", 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782573+0000", 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9700000000000005e-07 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782577+0000", 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8119999999999998e-06 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782587+0000", 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5140000000000007e-06 2024-04-24T00:38:17.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.782594+0000", 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.968e-06 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855774+0000", 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.073180178999999998 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855862+0000", 2024-04-24T00:38:17.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7793999999999993e-05 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:37:12.855964+0000", 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010183099999999999 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:175102 3.11 3:8ca8b224:::rbd_header.115ec7ed39de:head [watch reconnect cookie 140203704658000 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:17.666 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 13.936773397, 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98633510999999996, 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 175102 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:17.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635061+0000", 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635063+0000", 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.798e-06 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635070+0000", 2024-04-24T00:38:17.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4719999999999998e-06 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635088+0000", 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7853e-05 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635099+0000", 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1058e-05 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635160+0000", 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0868000000000001e-05 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:03.635172+0000", 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2459e-05 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.615851+0000", 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98067890000000002 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.616075+0000", 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022393399999999999 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:38:04.621396+0000", 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005320768 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:17.671 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:17.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:38:17.673 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:17.670+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:17.737 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T00:38:17.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T00:38:17.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T00:38:17.774 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:17.770+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:17.859 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:17.859 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:17.859 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-04-24T00:38:17.859 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:17.874 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:17.870+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:17.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:38:17.975 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:17.970+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:18.041 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:18.041 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-04-24T00:38:18.041 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-04-24T00:38:18.041 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-04-24T00:38:18.041 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:18.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:38:18.100 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:18.098+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:18.177 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:18.174+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:18.179 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4204.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 277.250585161, 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10184385999999999, 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.180 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4204", 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.167:0/3610312149", 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926406+0000", 2024-04-24T00:38:18.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926408+0000", 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6929999999999999e-06 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926422+0000", 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3973e-05 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926444+0000", 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1310000000000001e-05 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926461+0000", 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.694e-05 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926581+0000", 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120356 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:40.926595+0000", 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3867e-05 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025253+0000", 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098658598 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.025656+0000", 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040305700000000001 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0", 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.026017+0000", 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000361097 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.027292+0000", 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012741370000000001 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028091+0000", 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799867 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028183+0000", 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1955999999999998e-05 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:33:41.028249+0000", 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6008999999999998e-05 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10590 3.13 3:ce64de49:::rbd_data.115e380d621c.000000000000011a:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 244.31861947300001, 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055070651999999998, 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10590 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858371+0000", 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858375+0000", 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6500000000000002e-06 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858391+0000", 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.594e-05 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858429+0000", 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8219000000000001e-05 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858452+0000", 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3107000000000001e-05 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.858622+0000", 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000169388 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859008+0000", 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038648399999999998 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.862062+0000", 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030539819999999998 2024-04-24T00:38:18.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913107+0000", 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051045025000000001 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913197+0000", 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9841999999999998e-05 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913351+0000", 2024-04-24T00:38:18.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000153826 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.913442+0000", 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1188999999999997e-05 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:10592 3.16 3:6e800a72:::rbd_data.115e380d621c.000000000000011c:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 244.31737072499999, 2024-04-24T00:38:18.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.059773934000000001, 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 10592 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859620+0000", 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859622+0000", 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1969999999999999e-06 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859634+0000", 2024-04-24T00:38:18.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859662+0000", 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8274000000000001e-05 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859676+0000", 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4283e-05 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.859827+0000", 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015057800000000001 2024-04-24T00:38:18.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.860236+0000", 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00040954000000000001 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.865305+0000", 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005068159 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919120+0000", 2024-04-24T00:38:18.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053815533999999998 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919212+0000", 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1477999999999994e-05 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919315+0000", 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010359199999999999 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:13.919394+0000", 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8825999999999994e-05 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:20563 3.f 3:f33b3e83:::rbd_data.115ee4742374.000000000000000c:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 235.15974662100001, 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065595344, 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 20563 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017244+0000", 2024-04-24T00:38:18.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017246+0000", 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7600000000000001e-06 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017254+0000", 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1650000000000006e-06 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017269+0000", 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4591e-05 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017278+0000", 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5349999999999993e-06 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017353+0000", 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5304999999999998e-05 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.017445+0000", 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1598e-05 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4", 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.020330+0000", 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028853120000000001 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082723+0000", 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.062392610000000001 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082744+0000", 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0564000000000002e-05 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082793+0000", 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9505000000000001e-05 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082839+0000", 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6399000000000001e-05 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:20563 3.f e49/21 ondisk, result = 0)", 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 235.146252078, 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051994935999999999, 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:18.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030739+0000", 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030741+0000", 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8369999999999999e-06 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030750+0000", 2024-04-24T00:38:18.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0359999999999995e-06 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030780+0000", 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0300000000000001e-05 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.030796+0000", 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.588e-05 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082592+0000", 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.051796594000000001 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082604+0000", 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1251e-05 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:23.082734+0000", 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013003799999999999 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.207 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:21613 3.7 3:e1432e95:::rbd_data.115eeb9272a5.000000000000001d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 230.88447096300001, 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.052508659999999999, 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 21613 2024-04-24T00:38:18.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292520+0000", 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292521+0000", 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.161e-06 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292528+0000", 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8970000000000002e-06 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292543+0000", 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5464e-05 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292551+0000", 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5859999999999999e-06 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292610+0000", 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8755999999999998e-05 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.292678+0000", 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8337999999999995e-05 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.295248+0000", 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025701180000000001 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344637+0000", 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049389203 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344688+0000", 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0816000000000003e-05 2024-04-24T00:38:18.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.344848+0000", 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016021199999999999 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:27.345028+0000", 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018010899999999999 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22398 3.12 3:4eac7136:::rbd_header.115e6a380959:head [watch reconnect cookie 140203704480800 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.63088482000001, 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.018708084, 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22398 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.214 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546106+0000", 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546111+0000", 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3669999999999996e-06 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546141+0000", 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0015999999999999e-05 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546177+0000", 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5663000000000001e-05 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546201+0000", 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4223000000000002e-05 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546324+0000", 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123289 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.546332+0000", 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5329999999999997e-06 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553296+0000", 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069643939999999996 2024-04-24T00:38:18.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.553311+0000", 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4518999999999999e-05 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561392+0000", 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0080812400000001 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.561590+0000", 2024-04-24T00:38:18.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019771200000000001 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.564814+0000", 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032241280000000002 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:22403 3.12 3:4eac7136:::rbd_header.115e6a380959:head [call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.603210956, 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98916561999999997, 2024-04-24T00:38:18.219 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 22403 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573780+0000", 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573782+0000", 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5019999999999999e-06 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573791+0000", 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0710000000000001e-06 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573814+0000", 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2293000000000002e-05 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573826+0000", 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2605e-05 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573897+0000", 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1000999999999993e-05 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:36.573905+0000", 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7530000000000001e-06 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562438+0000", 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98853340999999995 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562671+0000", 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023254800000000001 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:37.562945+0000", 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027443699999999999 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:30134 3.7 3:e07aa1cb:::rbd_data.115e8210e51a.0000000000000038:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 217.109188136, 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.069562784000000003, 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.224 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30134 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067803+0000", 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067804+0000", 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067813+0000", 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2840000000000006e-06 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067832+0000", 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9638000000000001e-05 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067844+0000", 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1887e-05 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.067964+0000", 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119961 2024-04-24T00:38:18.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.068125+0000", 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016070599999999999 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.071628+0000", 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0035032610000000001 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137274+0000", 2024-04-24T00:38:18.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.065645588000000005 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137288+0000", 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3913e-05 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137327+0000", 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9718000000000002e-05 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137365+0000", 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8173000000000002e-05 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.4446.0:30134 3.7 e66/44 ondisk, result = 0)", 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 217.10017930199999, 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060469268, 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.230 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076811+0000", 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076813+0000", 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.105e-06 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076817+0000", 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1590000000000003e-06 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076829+0000", 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2439000000000001e-05 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.076839+0000", 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0179e-05 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137261+0000", 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.060421598 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137269+0000", 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6930000000000007e-06 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:34:41.137281+0000", 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2095e-05 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46415 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 368640~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.598480354, 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046552756000000001, 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.234 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46415 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578510+0000", 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578512+0000", 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-06 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578518+0000", 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4470000000000003e-06 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578536+0000", 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7873e-05 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578550+0000", 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3923e-05 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.623957+0000", 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045407227000000001 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.624126+0000", 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000168813 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625063+0000", 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00093727000000000003 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46416 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 385024~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.59841800800001, 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047640349999999998, 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.238 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46416 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578573+0000", 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578574+0000", 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7300000000000005e-07 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578578+0000", 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9169999999999999e-06 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578587+0000", 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7489999999999993e-06 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.578595+0000", 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6669999999999996e-06 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625153+0000", 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046558090000000003 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.625438+0000", 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000284787 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626213+0000", 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00077526699999999995 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46497 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 393216~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.242 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.59790832900001, 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047676981, 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46497 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:18.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579082+0000", 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579085+0000", 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1660000000000001e-06 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579094+0000", 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2539999999999998e-06 2024-04-24T00:38:18.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579135+0000", 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579153+0000", 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7816e-05 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626265+0000", 2024-04-24T00:38:18.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047111739 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626386+0000", 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120942 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626759+0000", 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037367500000000002 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46498 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 409600~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.597797267, 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048061668000000002, 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46498 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579193+0000", 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579195+0000", 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0169999999999999e-06 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579203+0000", 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7759999999999997e-06 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579223+0000", 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9233999999999999e-05 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579238+0000", 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5075999999999999e-05 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626863+0000", 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047625145000000001 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.626984+0000", 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121562 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627255+0000", 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027085800000000001 2024-04-24T00:38:18.250 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46499 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 425984~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.597732001, 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048355205999999998, 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46499 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579259+0000", 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579260+0000", 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3230000000000001e-06 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579266+0000", 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7069999999999997e-06 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579283+0000", 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7243000000000001e-05 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579296+0000", 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2986999999999999e-05 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627316+0000", 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048020162999999998 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627453+0000", 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013707300000000001 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627614+0000", 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016071 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46500 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 442368~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.59767193499999, 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048830170999999999, 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.255 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46500 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579319+0000", 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579320+0000", 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3e-06 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579326+0000", 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1199999999999999e-06 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579343+0000", 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7197000000000002e-05 2024-04-24T00:38:18.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579358+0000", 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4266e-05 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627848+0000", 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048490694000000001 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.627980+0000", 2024-04-24T00:38:18.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013134799999999999 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628149+0000", 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016924600000000001 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46501 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 458752~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.59761524199999, 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049941012999999999, 2024-04-24T00:38:18.259 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46501 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579376+0000", 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579377+0000", 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459999999999999e-06 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579390+0000", 2024-04-24T00:38:18.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.36e-05 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579407+0000", 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6747000000000001e-05 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579425+0000", 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8196999999999999e-05 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.628200+0000", 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.048774947999999999 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629093+0000", 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00089268699999999997 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629317+0000", 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022368799999999999 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46502 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 475136~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.59754590899999, 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050211595999999997, 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46502 2024-04-24T00:38:18.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579445+0000", 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579446+0000", 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1030000000000001e-06 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579452+0000", 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.874e-06 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579494+0000", 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2585e-05 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579510+0000", 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5736999999999999e-05 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629376+0000", 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049866133999999999 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629459+0000", 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2521000000000005e-05 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629656+0000", 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019764199999999999 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46503 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 491520~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.59745529, 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050483174999999998, 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.268 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46503 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579535+0000", 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579537+0000", 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.26e-06 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579543+0000", 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.613e-06 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579559+0000", 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5984000000000001e-05 2024-04-24T00:38:18.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579568+0000", 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629723+0000", 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050155376000000002 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.629857+0000", 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013409000000000001 2024-04-24T00:38:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630019+0000", 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161662 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.4446.0:46504 3.7 3:e020d460:::rbd_data.115ec7ed39de.0000000000000001:1d [read 507904~8192 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 197.59741187700001, 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050939640000000001, 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "continuous": false, 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-04-24T00:38:18.272 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.4446", 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46504 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-04-24T00:38:18.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579579+0000", 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579580+0000", 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1699999999999997e-07 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579583+0000", 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3459999999999998e-06 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579592+0000", 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2629999999999992e-06 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.579599+0000", 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1899999999999998e-06 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630094+0000", 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050494899000000003 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630207+0000", 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113112 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-04-24T00:35:00.630519+0000", 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031101300000000001 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-04-24T00:38:18.276 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:18.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:38:18.278 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-04-24T00:38:18.274+0000 7f2fa74a4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25214) UID: 0 2024-04-24T00:38:18.379 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-04-24T00:38:18.374+0000 7f5c6f834640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25218) UID: 0 2024-04-24T00:38:18.409 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:18.409 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:38:18.409 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:38:18.409 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:18.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:38:18.480 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-04-24T00:38:18.474+0000 7f82b44e3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25172) UID: 0 2024-04-24T00:38:18.550 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:38:18.550 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-04-24T00:38:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-04-24T00:38:18.551 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:18.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:38:18.609 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:18.606+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:18.681 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-04-24T00:38:18.678+0000 7f13008bb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33949) UID: 0 2024-04-24T00:38:18.732 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:18.732 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110941 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 32768~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.845838794, 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99518982899999997, 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110941 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882158+0000", 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882159+0000", 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882161+0000", 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1840000000000002e-06 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882169+0000", 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8480000000000001e-06 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882173+0000", 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6110000000000004e-06 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886136+0000", 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039630810000000002 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886147+0000", 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0484e-05 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876102+0000", 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98995489400000003 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.876434+0000", 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033270799999999997 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.877348+0000", 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091357700000000005 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110942 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 229376~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.845817048, 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99714419399999998, 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110942 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882180+0000", 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5199999999999998e-07 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882182+0000", 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8819999999999999e-06 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882190+0000", 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3629999999999996e-06 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882194+0000", 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5000000000000001e-06 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886176+0000", 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039815329999999998 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886184+0000", 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.038e-06 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878386+0000", 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99220202999999996 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.878535+0000", 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0001494 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879324+0000", 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078909599999999998 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110958 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 65536~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:38:18.742 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.845720736, 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.998192368, 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110958 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:38:18.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882276+0000", 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882277+0000", 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.15e-07 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882278+0000", 2024-04-24T00:38:18.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6419999999999999e-06 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882284+0000", 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3890000000000004e-06 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882288+0000", 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.138e-06 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886955+0000", 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046673289999999996 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886964+0000", 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8310000000000007e-06 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879623+0000", 2024-04-24T00:38:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99265892499999997 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.879818+0000", 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019486800000000001 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880469+0000", 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065093099999999999 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110959 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 262144~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.845660838, 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99945926699999998, 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110959 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.748 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882336+0000", 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882337+0000", 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4499999999999997e-07 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882340+0000", 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0259999999999999e-06 2024-04-24T00:38:18.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882358+0000", 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8689999999999999e-05 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882365+0000", 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3879999999999997e-06 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886989+0000", 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046244930000000004 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.886997+0000", 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880603+0000", 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99360585199999996 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.880779+0000", 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175792 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881795+0000", 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001016922 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110974 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 98304~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.753 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.845513393, 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.000617986, 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110974 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882484+0000", 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2100000000000002e-07 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882486+0000", 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.215e-06 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882494+0000", 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9410000000000007e-06 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882500+0000", 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2389999999999999e-06 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888427+0000", 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059265029999999996 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888436+0000", 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2019999999999993e-06 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.881936+0000", 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99350027399999996 2024-04-24T00:38:18.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.882146+0000", 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020958500000000001 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883102+0000", 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095560600000000001 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110975 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 294912~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.845489911, 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.002014129, 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110975 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882507+0000", 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882508+0000", 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9900000000000001e-07 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882510+0000", 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.148e-06 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882515+0000", 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7289999999999997e-06 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882520+0000", 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3290000000000004e-06 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888463+0000", 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059433669999999997 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.888470+0000", 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.187e-06 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883245+0000", 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99477520799999997 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.883497+0000", 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025132699999999997 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884521+0000", 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001024435 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110990 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 131072~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:38:18.763 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.84533877299999, 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0028161980000001, 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110990 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:38:18.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882658+0000", 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882659+0000", 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4299999999999998e-07 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882661+0000", 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-06 2024-04-24T00:38:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882667+0000", 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.879e-06 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882671+0000", 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5349999999999998e-06 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889926+0000", 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072543970000000001 2024-04-24T00:38:18.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889935+0000", 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7070000000000004e-06 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884669+0000", 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99473391700000002 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.884841+0000", 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017212600000000001 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885474+0000", 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063308099999999996 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:110991 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 327680~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.84531972000001, 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0041004259999999, 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110991 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882677+0000", 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882678+0000", 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882684+0000", 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3219999999999998e-06 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882691+0000", 2024-04-24T00:38:18.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4730000000000003e-06 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882705+0000", 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3813e-05 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889972+0000", 2024-04-24T00:38:18.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072669309999999999 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.889981+0000", 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3059999999999998e-06 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885568+0000", 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99558754299999996 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.885798+0000", 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023032799999999999 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886778+0000", 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097934600000000008 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111006 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 163840~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:38:18.773 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.845183928, 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0048379629999999, 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111006 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:38:18.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882813+0000", 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6100000000000002e-07 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882815+0000", 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7549999999999999e-06 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882822+0000", 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0539999999999997e-06 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882827+0000", 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3619999999999999e-06 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890715+0000", 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078878809999999994 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890723+0000", 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6289999999999999e-06 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.886854+0000", 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99613063000000002 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887032+0000", 2024-04-24T00:38:18.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017786200000000001 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887651+0000", 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061942900000000001 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111007 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:3e [read 360448~32768 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:38:18.778 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.84516449900001, 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061255339999999, 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111007 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882833+0000", 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0400000000000002e-07 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882835+0000", 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7129999999999999e-06 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882842+0000", 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1280000000000002e-06 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.882846+0000", 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5600000000000004e-06 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890749+0000", 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079023110000000004 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.890758+0000", 2024-04-24T00:38:18.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2070000000000001e-06 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.887756+0000", 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.99699837099999999 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888019+0000", 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000263014 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.888958+0000", 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093882600000000003 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111032 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.815230743, 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97659819800000003, 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111032 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912766+0000", 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912767+0000", 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2600000000000002e-07 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912769+0000", 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6759999999999999e-06 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912777+0000", 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7659999999999998e-06 2024-04-24T00:38:18.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.912782+0000", 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0590000000000002e-06 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913091+0000", 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030847099999999999 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.913101+0000", 2024-04-24T00:38:18.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0366999999999999e-05 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889077+0000", 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.975975437 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889228+0000", 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015131699999999999 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889364+0000", 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013657900000000001 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111065 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.804435148, 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96634889000000002, 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.788 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111065 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923562+0000", 2024-04-24T00:38:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923563+0000", 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7100000000000001e-07 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923566+0000", 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.658e-06 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923577+0000", 2024-04-24T00:38:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0268999999999999e-05 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923588+0000", 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1299e-05 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923668+0000", 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0012000000000006e-05 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.923678+0000", 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9019999999999994e-06 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889478+0000", 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96580058899999999 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889697+0000", 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218285 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889911+0000", 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000214105 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111116 3.1f 3:f85dfb34:::rbd_data.115e397a9cd4.000000000000000c:head [write 0~524288 [fadvise_sequential+fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.793233706, 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96849046100000002, 2024-04-24T00:38:18.793 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111116 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934763+0000", 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.934766+0000", 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3580000000000001e-06 2024-04-24T00:38:18.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935127+0000", 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036108199999999998 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935150+0000", 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2756999999999999e-05 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935159+0000", 2024-04-24T00:38:18.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6639999999999993e-06 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935249+0000", 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9470999999999998e-05 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935263+0000", 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4445e-05 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.889988+0000", 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.95472527200000001 2024-04-24T00:38:18.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890286+0000", 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029795199999999998 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 5", 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.890670+0000", 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038387799999999999 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902981+0000", 2024-04-24T00:38:18.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012310399 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903018+0000", 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6999000000000003e-05 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903151+0000", 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013311700000000001 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.903254+0000", 2024-04-24T00:38:18.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010306699999999999 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111119 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.792253761, 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96712422899999995, 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111119 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935743+0000", 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935744+0000", 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.206e-06 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935750+0000", 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.203e-06 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935761+0000", 2024-04-24T00:38:18.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1266e-05 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935766+0000", 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2830000000000001e-06 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935841+0000", 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4685999999999996e-05 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.935854+0000", 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3135e-05 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902400+0000", 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.96654596800000003 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902638+0000", 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023839799999999999 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.902868+0000", 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229084 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:111159 3.1f 3:fff8df7c:::rbd_data.115e1702e7a8.000000000000000b:snapdir [list-snaps] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.76329487700001, 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.94163226300000002, 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.805 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111159 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964702+0000", 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964704+0000", 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3850000000000001e-06 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964711+0000", 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4710000000000001e-06 2024-04-24T00:38:18.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964727+0000", 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6416e-05 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964736+0000", 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9209999999999998e-06 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964809+0000", 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3047999999999996e-05 2024-04-24T00:38:18.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:28.964821+0000", 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1738e-05 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.905869+0000", 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.941048035 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906136+0000", 2024-04-24T00:38:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026686400000000001 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:29.906334+0000", 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019838500000000001 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:143678 3.b 3:d70c067f:::rbd_data.115e8dbf22aa.000000000000005e:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:38:18.810 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 88.891065053999995, 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18713334100000001, 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143678 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836932+0000", 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836936+0000", 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9829999999999998e-06 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836953+0000", 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7016000000000001e-05 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.836986+0000", 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3319999999999999e-05 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837001+0000", 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5123000000000001e-05 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837183+0000", 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018131700000000001 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.837430+0000", 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024749599999999999 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:49.847946+0000", 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01051566 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023623+0000", 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17567688400000001 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023871+0000", 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024814800000000001 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.023984+0000", 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011326499999999999 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:36:50.024065+0000", 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1129000000000002e-05 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:173788 1.0 1:535385f0:::rbd_children:head [call rbd.get_children in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:44.817694+0000", 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.910303073000001, 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.79388813800000002, 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 173788 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:44.817694+0000", 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:44.817694+0000", 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:44.817695+0000", 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3999999999999999e-06 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:44.817702+0000", 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-06 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:44.817713+0000", 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0949e-05 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:44.817719+0000", 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7980000000000002e-06 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:44.817781+0000", 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2855999999999996e-05 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:44.817792+0000", 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0181e-05 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.611286+0000", 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.79349472300000001 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:45.611582+0000", 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000295801 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:173913 3.12 3:4e045fb2:::rbd_data.115e6f07caf6.0000000000000007:head [delete] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:49.667236+0000", 2024-04-24T00:38:18.820 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.060760893000001, 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.98316133500000003, 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 173913 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.667236+0000", 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.667236+0000", 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.667237+0000", 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0900000000000001e-07 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.667240+0000", 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4319999999999999e-06 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.667248+0000", 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0520000000000002e-06 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.667255+0000", 2024-04-24T00:38:18.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4779999999999996e-06 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.667319+0000", 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4219000000000003e-05 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.667330+0000", 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0818e-05 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.645115+0000", 2024-04-24T00:38:18.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97778521900000004 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.645236+0000", 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012075599999999999 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.645335+0000", 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9074999999999996e-05 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.648008+0000", 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026734459999999999 2024-04-24T00:38:18.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.650159+0000", 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021504409999999999 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.650211+0000", 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1975000000000003e-05 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.650397+0000", 2024-04-24T00:38:18.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186715 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:173921 3.12 3:4d82fa19:::rbd_data.115e6f07caf6.0000000000000011:head [delete] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:37:49.675387+0000", 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.052610345000002, 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97522372199999996, 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 173921 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.675387+0000", 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.675387+0000", 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.675388+0000", 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.181e-06 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.675393+0000", 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2270000000000001e-06 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.675407+0000", 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3667e-05 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.675414+0000", 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5669999999999997e-06 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.675479+0000", 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4413999999999997e-05 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:49.675488+0000", 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7499999999999998e-06 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.646361+0000", 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97087277100000002 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.646477+0000", 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011532 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3", 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.646573+0000", 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5982000000000004e-05 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.650408+0000", 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003834952 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.650443+0000", 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5896999999999999e-05 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.650483+0000", 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9943e-05 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:37:50.650610+0000", 2024-04-24T00:38:18.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012705099999999999 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4446.0:175103 3.f 3:f72043c6:::rbd_object_map.115ec7ed39de:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-04-24T00:38:03.653276+0000", 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.074720612, 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97997563700000001, 2024-04-24T00:38:18.833 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4446", 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/2320043960", 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 175103 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-04-24T00:38:18.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.653276+0000", 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.653276+0000", 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.653280+0000", 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.315e-06 2024-04-24T00:38:18.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.653295+0000", 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5085e-05 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.653338+0000", 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2688000000000002e-05 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.653359+0000", 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1889e-05 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.653462+0000", 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010223800000000001 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:03.653476+0000", 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3995999999999999e-05 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:04.625608+0000", 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.97213194000000003 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:04.626070+0000", 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046266100000000001 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:04.626857+0000", 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078658600000000003 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:04.629372+0000", 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025148810000000001 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:04.633053+0000", 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036815239999999998 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:04.633166+0000", 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011237499999999999 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-04-24T00:38:04.633252+0000", 2024-04-24T00:38:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6458999999999993e-05 2024-04-24T00:38:18.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-24T00:38:18.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-24T00:38:18.840 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:18.842 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-04-24T00:38:18.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T00:38:18.843 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T00:38:18.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-24T00:38:18.844 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-04-24T00:38:18.838+0000 7f36ab9c4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25170) UID: 0 2024-04-24T00:38:18.917 INFO:teuthology.orchestra.run.smithi064.stdout:All operations completed A-OK! 2024-04-24T00:38:18.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T00:38:18.970 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-04-24T00:38:18.966+0000 7fd7a6389640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25220) UID: 0 2024-04-24T00:38:18.994 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:18.994 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-04-24T00:38:19.004 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-24T00:38:19.026 INFO:tasks.thrashosds:joining thrashosds 2024-04-24T00:38:19.027 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-04-24T00:38:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-04-24T00:38:19.027 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:19.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T00:38:19.113 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-04-24T00:38:19.113 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T00:38:19.131 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-04-24T00:38:19.132 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T00:38:19.149 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-04-24T00:38:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T00:38:19.166 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-04-24T00:38:19.166 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T00:38:19.187 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-04-24T00:38:19.187 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T00:38:19.200 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:19.200 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:19.207 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-04-24T00:38:19.208 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T00:38:19.215 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T00:38:19.229 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-04-24T00:38:19.229 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-24T00:38:19.250 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-04-24T00:38:19.250 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-24T00:38:19.357 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:19.357 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:19.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T00:38:19.500 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:19.501 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:19.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T00:38:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-04-24T00:38:19.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T00:38:19.638 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:19.638 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:19.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T00:38:19.823 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:19.823 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:19.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T00:38:19.960 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:19.960 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:19.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T00:38:20.105 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:20.105 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:20.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T00:38:20.246 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:20.247 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:20.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T00:38:20.394 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:20.394 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:20.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T00:38:20.538 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:20.538 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:20.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T00:38:20.676 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:20.676 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:20.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T00:38:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T00:38:20.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T00:38:20.813 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:20.813 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:20.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T00:38:20.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T00:38:20.950 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:20.950 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:20.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T00:38:21.081 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:21.081 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:21.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-24T00:38:21.217 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:21.217 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:21.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-24T00:38:21.353 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:21.353 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:21.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-24T00:38:21.490 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:21.490 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:21.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-24T00:38:21.628 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:21.629 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:21.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-24T00:38:21.764 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:21.764 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:21.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-24T00:38:21.782 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T00:38:21.782 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-04-24T00:38:21.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T00:38:21.812 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T00:38:21.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:38:21.899 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:21.900 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:21.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T00:38:22.037 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:22.037 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:22.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T00:38:22.184 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:22.184 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:22.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T00:38:22.200 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:22.200 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:38:22.220 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-04-24T00:38:21.784398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6401223,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":185649,"num_read_kb":1545557,"num_write":31352,"num_write_kb":1020585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6900,"num_bytes_recovered":359846155,"num_keys_recovered":554,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22916,"ondisk_log_size":22916,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":368016,"kb_used_data":19108,"kb_used_omap":14,"kb_used_meta":348721,"kb_avail":565863024,"statfs":{"total":579820584960,"available":579443736576,"internally_reserved":0,"allocated":19566592,"data_stored":11025826,"data_compressed":103730,"data_compressed_allocated":180224,"data_compressed_original":1388544,"omap_allocated":14401,"internal_metadata":357091263},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-249937195,"num_objects":-157,"num_object_clones":0,"num_object_copies":-314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-157,"num_whiteouts":0,"num_read":1090,"num_read_kb":937,"num_write":322,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043124"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":5954,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803340+0000","last_change":"2024-04-24T00:38:18.803340+0000","last_active":"2024-04-24T00:38:18.803340+0000","last_peered":"2024-04-24T00:38:18.803340+0000","last_clean":"2024-04-24T00:38:18.803340+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:18.803340+0000","last_undegraded":"2024-04-24T00:38:18.803340+0000","last_fullsized":"2024-04-24T00:38:18.803340+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:16.140487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1e","version":"264'1050","reported_seq":5667,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.755961+0000","last_change":"2024-04-24T00:38:17.755961+0000","last_active":"2024-04-24T00:38:17.755961+0000","last_peered":"2024-04-24T00:38:17.755961+0000","last_clean":"2024-04-24T00:38:17.755961+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:38:17.755961+0000","last_undegraded":"2024-04-24T00:38:17.755961+0000","last_fullsized":"2024-04-24T00:38:17.755961+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:50.248491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020495069999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1d","version":"263'591","reported_seq":7253,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.768461+0000","last_change":"2024-04-24T00:38:18.768970+0000","last_active":"2024-04-24T00:38:18.768461+0000","last_peered":"2024-04-24T00:38:18.768461+0000","last_clean":"2024-04-24T00:38:18.768461+0000","last_became_active":"2024-04-24T00:38:03.632430+0000","last_became_peered":"2024-04-24T00:38:03.632430+0000","last_unstale":"2024-04-24T00:38:18.768461+0000","last_undegraded":"2024-04-24T00:38:18.768461+0000","last_fullsized":"2024-04-24T00:38:18.768461+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:57.114511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1c","version":"265'794","reported_seq":6031,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.774862+0000","last_change":"2024-04-24T00:38:18.768583+0000","last_active":"2024-04-24T00:38:18.774862+0000","last_peered":"2024-04-24T00:38:18.774862+0000","last_clean":"2024-04-24T00:38:18.774862+0000","last_became_active":"2024-04-24T00:38:04.608623+0000","last_became_peered":"2024-04-24T00:38:04.608623+0000","last_unstale":"2024-04-24T00:38:18.774862+0000","last_undegraded":"2024-04-24T00:38:18.774862+0000","last_fullsized":"2024-04-24T00:38:18.774862+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:01.877528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5263,"num_read_kb":61365,"num_write":1006,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1b","version":"264'498","reported_seq":4511,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.763948+0000","last_change":"2024-04-24T00:38:18.764647+0000","last_active":"2024-04-24T00:38:18.763948+0000","last_peered":"2024-04-24T00:38:18.763948+0000","last_clean":"2024-04-24T00:38:18.763948+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:18.763948+0000","last_undegraded":"2024-04-24T00:38:18.763948+0000","last_fullsized":"2024-04-24T00:38:18.763948+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:59.132716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1a","version":"265'390","reported_seq":6257,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.920724+0000","last_change":"2024-04-24T00:38:17.777890+0000","last_active":"2024-04-24T00:38:17.920724+0000","last_peered":"2024-04-24T00:38:17.920724+0000","last_clean":"2024-04-24T00:38:17.920724+0000","last_became_active":"2024-04-24T00:38:04.613035+0000","last_became_peered":"2024-04-24T00:38:04.613035+0000","last_unstale":"2024-04-24T00:38:17.920724+0000","last_undegraded":"2024-04-24T00:38:17.920724+0000","last_fullsized":"2024-04-24T00:38:17.920724+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:00:37.609101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001236025,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4505,"num_read_kb":37326,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":3953397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.19","version":"265'712","reported_seq":5857,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.778045+0000","last_change":"2024-04-24T00:38:17.778045+0000","last_active":"2024-04-24T00:38:17.778045+0000","last_peered":"2024-04-24T00:38:17.778045+0000","last_clean":"2024-04-24T00:38:17.778045+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:17.778045+0000","last_undegraded":"2024-04-24T00:38:17.778045+0000","last_fullsized":"2024-04-24T00:38:17.778045+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:04.494142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001300421,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.18","version":"263'551","reported_seq":4881,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.802371+0000","last_change":"2024-04-24T00:38:18.802601+0000","last_active":"2024-04-24T00:38:18.802371+0000","last_peered":"2024-04-24T00:38:18.802371+0000","last_clean":"2024-04-24T00:38:18.802371+0000","last_became_active":"2024-04-24T00:37:46.608136+0000","last_became_peered":"2024-04-24T00:37:46.608136+0000","last_unstale":"2024-04-24T00:38:18.802371+0000","last_undegraded":"2024-04-24T00:38:18.802371+0000","last_fullsized":"2024-04-24T00:38:18.802371+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:06.579921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080020700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.17","version":"263'339","reported_seq":4768,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803412+0000","last_change":"2024-04-24T00:38:18.803619+0000","last_active":"2024-04-24T00:38:18.803412+0000","last_peered":"2024-04-24T00:38:18.803412+0000","last_clean":"2024-04-24T00:38:18.803412+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:18.803412+0000","last_undegraded":"2024-04-24T00:38:18.803412+0000","last_fullsized":"2024-04-24T00:38:18.803412+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:53.921153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.16","version":"266'737","reported_seq":6716,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.785437+0000","last_change":"2024-04-24T00:38:18.756271+0000","last_active":"2024-04-24T00:38:18.785437+0000","last_peered":"2024-04-24T00:38:18.785437+0000","last_clean":"2024-04-24T00:38:18.785437+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:18.785437+0000","last_undegraded":"2024-04-24T00:38:18.785437+0000","last_fullsized":"2024-04-24T00:38:18.785437+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:36.855739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7572,"num_read_kb":50907,"num_write":1080,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.15","version":"264'632","reported_seq":7225,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.755919+0000","last_change":"2024-04-24T00:38:18.756263+0000","last_active":"2024-04-24T00:38:18.755919+0000","last_peered":"2024-04-24T00:38:18.755919+0000","last_clean":"2024-04-24T00:38:18.755919+0000","last_became_active":"2024-04-24T00:38:03.644911+0000","last_became_peered":"2024-04-24T00:38:03.644911+0000","last_unstale":"2024-04-24T00:38:18.755919+0000","last_undegraded":"2024-04-24T00:38:18.755919+0000","last_fullsized":"2024-04-24T00:38:18.755919+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:24:38.830943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.14","version":"264'496","reported_seq":5356,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.756067+0000","last_change":"2024-04-24T00:38:17.756319+0000","last_active":"2024-04-24T00:38:17.756067+0000","last_peered":"2024-04-24T00:38:17.756067+0000","last_clean":"2024-04-24T00:38:17.756067+0000","last_became_active":"2024-04-24T00:38:03.644711+0000","last_became_peered":"2024-04-24T00:38:03.644711+0000","last_unstale":"2024-04-24T00:38:17.756067+0000","last_undegraded":"2024-04-24T00:38:17.756067+0000","last_fullsized":"2024-04-24T00:38:17.756067+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:18.036694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022892910000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":508,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.754260+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:38:17.754260+0000","last_peered":"2024-04-24T00:38:17.754260+0000","last_clean":"2024-04-24T00:38:17.754260+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:38:17.754260+0000","last_undegraded":"2024-04-24T00:38:17.754260+0000","last_fullsized":"2024-04-24T00:38:17.754260+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6478,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803861+0000","last_change":"2024-04-24T00:38:18.803861+0000","last_active":"2024-04-24T00:38:18.803861+0000","last_peered":"2024-04-24T00:38:18.803861+0000","last_clean":"2024-04-24T00:38:18.803861+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:18.803861+0000","last_undegraded":"2024-04-24T00:38:18.803861+0000","last_fullsized":"2024-04-24T00:38:18.803861+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:52.120677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":68,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.777073+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:38:17.777073+0000","last_peered":"2024-04-24T00:38:17.777073+0000","last_clean":"2024-04-24T00:38:17.777073+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:17.777073+0000","last_undegraded":"2024-04-24T00:38:17.777073+0000","last_fullsized":"2024-04-24T00:38:17.777073+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4790,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803322+0000","last_change":"2024-04-24T00:38:18.803526+0000","last_active":"2024-04-24T00:38:18.803322+0000","last_peered":"2024-04-24T00:38:18.803322+0000","last_clean":"2024-04-24T00:38:18.803322+0000","last_became_active":"2024-04-24T00:37:47.617077+0000","last_became_peered":"2024-04-24T00:37:47.617077+0000","last_unstale":"2024-04-24T00:38:18.803322+0000","last_undegraded":"2024-04-24T00:38:18.803322+0000","last_fullsized":"2024-04-24T00:38:18.803322+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:04:25.048788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001038757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.801659+0000","last_change":"2024-04-24T00:38:00.615867+0000","last_active":"2024-04-24T00:38:18.801659+0000","last_peered":"2024-04-24T00:38:18.801659+0000","last_clean":"2024-04-24T00:38:18.801659+0000","last_became_active":"2024-04-24T00:38:00.615083+0000","last_became_peered":"2024-04-24T00:38:00.615083+0000","last_unstale":"2024-04-24T00:38:18.801659+0000","last_undegraded":"2024-04-24T00:38:18.801659+0000","last_fullsized":"2024-04-24T00:38:18.801659+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:45:44.557767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4109,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.777533+0000","last_change":"2024-04-24T00:38:17.777533+0000","last_active":"2024-04-24T00:38:17.777533+0000","last_peered":"2024-04-24T00:38:17.777533+0000","last_clean":"2024-04-24T00:38:17.777533+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:17.777533+0000","last_undegraded":"2024-04-24T00:38:17.777533+0000","last_fullsized":"2024-04-24T00:38:17.777533+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:16:58.702216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077823599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":68,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.776513+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:38:17.776513+0000","last_peered":"2024-04-24T00:38:17.776513+0000","last_clean":"2024-04-24T00:38:17.776513+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:17.776513+0000","last_undegraded":"2024-04-24T00:38:17.776513+0000","last_fullsized":"2024-04-24T00:38:17.776513+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6246,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.766784+0000","last_change":"2024-04-24T00:38:18.767450+0000","last_active":"2024-04-24T00:38:18.766784+0000","last_peered":"2024-04-24T00:38:18.766784+0000","last_clean":"2024-04-24T00:38:18.766784+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:18.766784+0000","last_undegraded":"2024-04-24T00:38:18.766784+0000","last_fullsized":"2024-04-24T00:38:18.766784+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:09:34.691573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.3","version":"266'580","reported_seq":6755,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.916313+0000","last_change":"2024-04-24T00:38:18.802265+0000","last_active":"2024-04-24T00:38:18.916313+0000","last_peered":"2024-04-24T00:38:18.916313+0000","last_clean":"2024-04-24T00:38:18.916313+0000","last_became_active":"2024-04-24T00:38:04.615801+0000","last_became_peered":"2024-04-24T00:38:04.615801+0000","last_unstale":"2024-04-24T00:38:18.916313+0000","last_undegraded":"2024-04-24T00:38:18.916313+0000","last_fullsized":"2024-04-24T00:38:18.916313+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:03:27.537917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":435,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.762797+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:38:18.762797+0000","last_peered":"2024-04-24T00:38:18.762797+0000","last_clean":"2024-04-24T00:38:18.762797+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:18.762797+0000","last_undegraded":"2024-04-24T00:38:18.762797+0000","last_fullsized":"2024-04-24T00:38:18.762797+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7923,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.778133+0000","last_change":"2024-04-24T00:38:17.778390+0000","last_active":"2024-04-24T00:38:17.778133+0000","last_peered":"2024-04-24T00:38:17.778133+0000","last_clean":"2024-04-24T00:38:17.778133+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:17.778133+0000","last_undegraded":"2024-04-24T00:38:17.778133+0000","last_fullsized":"2024-04-24T00:38:17.778133+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:20.034317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014884779999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":503,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.763233+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:38:17.763233+0000","last_peered":"2024-04-24T00:38:17.763233+0000","last_clean":"2024-04-24T00:38:17.763233+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:17.763233+0000","last_undegraded":"2024-04-24T00:38:17.763233+0000","last_fullsized":"2024-04-24T00:38:17.763233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":4753,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.802150+0000","last_change":"2024-04-24T00:38:18.802387+0000","last_active":"2024-04-24T00:38:18.802150+0000","last_peered":"2024-04-24T00:38:18.802150+0000","last_clean":"2024-04-24T00:38:18.802150+0000","last_became_active":"2024-04-24T00:38:03.642971+0000","last_became_peered":"2024-04-24T00:38:03.642971+0000","last_unstale":"2024-04-24T00:38:18.802150+0000","last_undegraded":"2024-04-24T00:38:18.802150+0000","last_fullsized":"2024-04-24T00:38:18.802150+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:52.299140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.776488+0000","last_change":"2024-04-24T00:37:48.620512+0000","last_active":"2024-04-24T00:38:17.776488+0000","last_peered":"2024-04-24T00:38:17.776488+0000","last_clean":"2024-04-24T00:38:17.776488+0000","last_became_active":"2024-04-24T00:37:48.620159+0000","last_became_peered":"2024-04-24T00:37:48.620159+0000","last_unstale":"2024-04-24T00:38:17.776488+0000","last_undegraded":"2024-04-24T00:38:17.776488+0000","last_fullsized":"2024-04-24T00:38:17.776488+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:47.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2855,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.767788+0000","last_change":"2024-04-24T00:38:18.768267+0000","last_active":"2024-04-24T00:38:18.767788+0000","last_peered":"2024-04-24T00:38:18.767788+0000","last_clean":"2024-04-24T00:38:18.767788+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:38:18.767788+0000","last_undegraded":"2024-04-24T00:38:18.767788+0000","last_fullsized":"2024-04-24T00:38:18.767788+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:30:41.411539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049969319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":435,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.762994+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:38:18.762994+0000","last_peered":"2024-04-24T00:38:18.762994+0000","last_clean":"2024-04-24T00:38:18.762994+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:18.762994+0000","last_undegraded":"2024-04-24T00:38:18.762994+0000","last_fullsized":"2024-04-24T00:38:18.762994+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":463,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.801523+0000","last_change":"2024-04-24T00:37:45.610754+0000","last_active":"2024-04-24T00:38:18.801523+0000","last_peered":"2024-04-24T00:38:18.801523+0000","last_clean":"2024-04-24T00:38:18.801523+0000","last_became_active":"2024-04-24T00:37:45.609794+0000","last_became_peered":"2024-04-24T00:37:45.609794+0000","last_unstale":"2024-04-24T00:38:18.801523+0000","last_undegraded":"2024-04-24T00:38:18.801523+0000","last_fullsized":"2024-04-24T00:38:18.801523+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:27.857798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6939,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.755173+0000","last_change":"2024-04-24T00:38:18.755623+0000","last_active":"2024-04-24T00:38:18.755173+0000","last_peered":"2024-04-24T00:38:18.755173+0000","last_clean":"2024-04-24T00:38:18.755173+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:18.755173+0000","last_undegraded":"2024-04-24T00:38:18.755173+0000","last_fullsized":"2024-04-24T00:38:18.755173+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:44.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.9","version":"265'987","reported_seq":5284,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.765132+0000","last_change":"2024-04-24T00:38:17.765132+0000","last_active":"2024-04-24T00:38:17.765132+0000","last_peered":"2024-04-24T00:38:17.765132+0000","last_clean":"2024-04-24T00:38:17.765132+0000","last_became_active":"2024-04-24T00:38:04.612514+0000","last_became_peered":"2024-04-24T00:38:04.612514+0000","last_unstale":"2024-04-24T00:38:17.765132+0000","last_undegraded":"2024-04-24T00:38:17.765132+0000","last_fullsized":"2024-04-24T00:38:17.765132+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.862820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.a","version":"263'1904","reported_seq":4533,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.755501+0000","last_change":"2024-04-24T00:38:17.755920+0000","last_active":"2024-04-24T00:38:17.755501+0000","last_peered":"2024-04-24T00:38:17.755501+0000","last_clean":"2024-04-24T00:38:17.755501+0000","last_became_active":"2024-04-24T00:38:05.613032+0000","last_became_peered":"2024-04-24T00:38:05.613032+0000","last_unstale":"2024-04-24T00:38:17.755501+0000","last_undegraded":"2024-04-24T00:38:17.755501+0000","last_fullsized":"2024-04-24T00:38:17.755501+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:57:47.180836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022763219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.b","version":"263'461","reported_seq":4299,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.802498+0000","last_change":"2024-04-24T00:38:18.802777+0000","last_active":"2024-04-24T00:38:18.802498+0000","last_peered":"2024-04-24T00:38:18.802498+0000","last_clean":"2024-04-24T00:38:18.802498+0000","last_became_active":"2024-04-24T00:37:45.607329+0000","last_became_peered":"2024-04-24T00:37:45.607329+0000","last_unstale":"2024-04-24T00:38:18.802498+0000","last_undegraded":"2024-04-24T00:38:18.802498+0000","last_fullsized":"2024-04-24T00:38:18.802498+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:25:17.095603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.c","version":"266'821","reported_seq":10059,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.766586+0000","last_change":"2024-04-24T00:38:18.766586+0000","last_active":"2024-04-24T00:38:18.766586+0000","last_peered":"2024-04-24T00:38:18.766586+0000","last_clean":"2024-04-24T00:38:18.766586+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:18.766586+0000","last_undegraded":"2024-04-24T00:38:18.766586+0000","last_fullsized":"2024-04-24T00:38:18.766586+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":821,"log_dups_size":0,"ondisk_log_size":821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:15.463491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":2786689,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10576,"num_read_kb":68776,"num_write":1168,"num_write_kb":35215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.d","version":"263'426","reported_seq":2911,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.766201+0000","last_change":"2024-04-24T00:38:18.766868+0000","last_active":"2024-04-24T00:38:18.766201+0000","last_peered":"2024-04-24T00:38:18.766201+0000","last_clean":"2024-04-24T00:38:18.766201+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:38:18.766201+0000","last_undegraded":"2024-04-24T00:38:18.766201+0000","last_fullsized":"2024-04-24T00:38:18.766201+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:24.780179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.e","version":"262'670","reported_seq":8055,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.778285+0000","last_change":"2024-04-24T00:38:17.779083+0000","last_active":"2024-04-24T00:38:17.778285+0000","last_peered":"2024-04-24T00:38:17.778285+0000","last_clean":"2024-04-24T00:38:17.778285+0000","last_became_active":"2024-04-24T00:38:03.647425+0000","last_became_peered":"2024-04-24T00:38:03.647425+0000","last_unstale":"2024-04-24T00:38:17.778285+0000","last_undegraded":"2024-04-24T00:38:17.778285+0000","last_fullsized":"2024-04-24T00:38:17.778285+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:33:42.482648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001649331,"stat_sum":{"num_bytes":3024147,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":951,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.f","version":"263'874","reported_seq":5861,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803001+0000","last_change":"2024-04-24T00:38:18.803247+0000","last_active":"2024-04-24T00:38:18.803001+0000","last_peered":"2024-04-24T00:38:18.803001+0000","last_clean":"2024-04-24T00:38:18.803001+0000","last_became_active":"2024-04-24T00:38:04.617797+0000","last_became_peered":"2024-04-24T00:38:04.617797+0000","last_unstale":"2024-04-24T00:38:18.803001+0000","last_undegraded":"2024-04-24T00:38:18.803001+0000","last_fullsized":"2024-04-24T00:38:18.803001+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:33:23.694830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.10","version":"265'610","reported_seq":6374,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.885339+0000","last_change":"2024-04-24T00:38:17.755366+0000","last_active":"2024-04-24T00:38:17.885339+0000","last_peered":"2024-04-24T00:38:17.885339+0000","last_clean":"2024-04-24T00:38:17.885339+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:17.885339+0000","last_undegraded":"2024-04-24T00:38:17.885339+0000","last_fullsized":"2024-04-24T00:38:17.885339+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:16:28.900971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001829174,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.11","version":"265'598","reported_seq":8447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.869475+0000","last_change":"2024-04-24T00:38:17.755488+0000","last_active":"2024-04-24T00:38:17.869475+0000","last_peered":"2024-04-24T00:38:17.869475+0000","last_clean":"2024-04-24T00:38:17.869475+0000","last_became_active":"2024-04-24T00:38:04.614076+0000","last_became_peered":"2024-04-24T00:38:04.614076+0000","last_unstale":"2024-04-24T00:38:17.869475+0000","last_undegraded":"2024-04-24T00:38:17.869475+0000","last_fullsized":"2024-04-24T00:38:17.869475+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:19:50.707765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.12","version":"265'1515","reported_seq":6097,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.801822+0000","last_change":"2024-04-24T00:38:18.802080+0000","last_active":"2024-04-24T00:38:18.801822+0000","last_peered":"2024-04-24T00:38:18.801822+0000","last_clean":"2024-04-24T00:38:18.801822+0000","last_became_active":"2024-04-24T00:37:52.611238+0000","last_became_peered":"2024-04-24T00:37:52.611238+0000","last_unstale":"2024-04-24T00:38:18.801822+0000","last_undegraded":"2024-04-24T00:38:18.801822+0000","last_fullsized":"2024-04-24T00:38:18.801822+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:46:09.244438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.13","version":"263'651","reported_seq":5048,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.763982+0000","last_change":"2024-04-24T00:38:17.764335+0000","last_active":"2024-04-24T00:38:17.763982+0000","last_peered":"2024-04-24T00:38:17.763982+0000","last_clean":"2024-04-24T00:38:17.763982+0000","last_became_active":"2024-04-24T00:38:04.612883+0000","last_became_peered":"2024-04-24T00:38:04.612883+0000","last_unstale":"2024-04-24T00:38:17.763982+0000","last_undegraded":"2024-04-24T00:38:17.763982+0000","last_fullsized":"2024-04-24T00:38:17.763982+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:40:45.891223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001199851,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5810855,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":185533,"num_read_kb":1545457,"num_write":31193,"num_write_kb":1018169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6894,"num_bytes_recovered":358075051,"num_keys_recovered":554,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5619712,"data_stored":36462251,"data_compressed":1807200,"data_compressed_allocated":4071424,"data_compressed_original":35677204,"omap_allocated":4834,"internal_metadata":0},"log_size":22799,"ondisk_log_size":22799,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":34064,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574909,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66296,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305544,"statfs":{"total":96636764160,"available":96568877056,"internally_reserved":0,"allocated":3256320,"data_stored":2191242,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70440,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301400,"statfs":{"total":96636764160,"available":96564633600,"internally_reserved":0,"allocated":3235840,"data_stored":1626319,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574908,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74596,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297244,"statfs":{"total":96636764160,"available":96560377856,"internally_reserved":0,"allocated":3231744,"data_stored":1626303,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574909,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60812,"kb_used_data":3264,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311028,"statfs":{"total":96636764160,"available":96574492672,"internally_reserved":0,"allocated":3342336,"data_stored":1747857,"data_compressed":89589,"data_compressed_allocated":94208,"data_compressed_original":135168,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60784,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311056,"statfs":{"total":96636764160,"available":96574521344,"internally_reserved":0,"allocated":3231744,"data_stored":1636410,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893158,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35088,"kb_used_data":3192,"kb_used_omap":6,"kb_used_meta":31865,"kb_avail":94336752,"statfs":{"total":96636764160,"available":96600834048,"internally_reserved":0,"allocated":3268608,"data_stored":2197695,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":32630499},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":35536,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":25429,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1863680,"data_stored":11425356,"data_compressed":520515,"data_compressed_allocated":1241088,"data_compressed_original":11121490,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":25445,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:38:22.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:38:22.312 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-04-24T00:38:22.313 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T00:38:22.560 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:22.561 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:38:22.579 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-04-24T00:38:21.784398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6401223,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":185649,"num_read_kb":1545557,"num_write":31352,"num_write_kb":1020585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6900,"num_bytes_recovered":359846155,"num_keys_recovered":554,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22916,"ondisk_log_size":22916,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":368016,"kb_used_data":19108,"kb_used_omap":14,"kb_used_meta":348721,"kb_avail":565863024,"statfs":{"total":579820584960,"available":579443736576,"internally_reserved":0,"allocated":19566592,"data_stored":11025826,"data_compressed":103730,"data_compressed_allocated":180224,"data_compressed_original":1388544,"omap_allocated":14401,"internal_metadata":357091263},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-249937195,"num_objects":-157,"num_object_clones":0,"num_object_copies":-314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-157,"num_whiteouts":0,"num_read":1090,"num_read_kb":937,"num_write":322,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043124"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":5954,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803340+0000","last_change":"2024-04-24T00:38:18.803340+0000","last_active":"2024-04-24T00:38:18.803340+0000","last_peered":"2024-04-24T00:38:18.803340+0000","last_clean":"2024-04-24T00:38:18.803340+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:18.803340+0000","last_undegraded":"2024-04-24T00:38:18.803340+0000","last_fullsized":"2024-04-24T00:38:18.803340+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:16.140487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1e","version":"264'1050","reported_seq":5667,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.755961+0000","last_change":"2024-04-24T00:38:17.755961+0000","last_active":"2024-04-24T00:38:17.755961+0000","last_peered":"2024-04-24T00:38:17.755961+0000","last_clean":"2024-04-24T00:38:17.755961+0000","last_became_active":"2024-04-24T00:34:17.648291+0000","last_became_peered":"2024-04-24T00:34:17.648291+0000","last_unstale":"2024-04-24T00:38:17.755961+0000","last_undegraded":"2024-04-24T00:38:17.755961+0000","last_fullsized":"2024-04-24T00:38:17.755961+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:50.248491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020495069999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1d","version":"263'591","reported_seq":7253,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.768461+0000","last_change":"2024-04-24T00:38:18.768970+0000","last_active":"2024-04-24T00:38:18.768461+0000","last_peered":"2024-04-24T00:38:18.768461+0000","last_clean":"2024-04-24T00:38:18.768461+0000","last_became_active":"2024-04-24T00:38:03.632430+0000","last_became_peered":"2024-04-24T00:38:03.632430+0000","last_unstale":"2024-04-24T00:38:18.768461+0000","last_undegraded":"2024-04-24T00:38:18.768461+0000","last_fullsized":"2024-04-24T00:38:18.768461+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:57.114511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1c","version":"265'794","reported_seq":6031,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.774862+0000","last_change":"2024-04-24T00:38:18.768583+0000","last_active":"2024-04-24T00:38:18.774862+0000","last_peered":"2024-04-24T00:38:18.774862+0000","last_clean":"2024-04-24T00:38:18.774862+0000","last_became_active":"2024-04-24T00:38:04.608623+0000","last_became_peered":"2024-04-24T00:38:04.608623+0000","last_unstale":"2024-04-24T00:38:18.774862+0000","last_undegraded":"2024-04-24T00:38:18.774862+0000","last_fullsized":"2024-04-24T00:38:18.774862+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:01.877528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5263,"num_read_kb":61365,"num_write":1006,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1b","version":"264'498","reported_seq":4511,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.763948+0000","last_change":"2024-04-24T00:38:18.764647+0000","last_active":"2024-04-24T00:38:18.763948+0000","last_peered":"2024-04-24T00:38:18.763948+0000","last_clean":"2024-04-24T00:38:18.763948+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:18.763948+0000","last_undegraded":"2024-04-24T00:38:18.763948+0000","last_fullsized":"2024-04-24T00:38:18.763948+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:59.132716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.1a","version":"265'390","reported_seq":6257,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.920724+0000","last_change":"2024-04-24T00:38:17.777890+0000","last_active":"2024-04-24T00:38:17.920724+0000","last_peered":"2024-04-24T00:38:17.920724+0000","last_clean":"2024-04-24T00:38:17.920724+0000","last_became_active":"2024-04-24T00:38:04.613035+0000","last_became_peered":"2024-04-24T00:38:04.613035+0000","last_unstale":"2024-04-24T00:38:17.920724+0000","last_undegraded":"2024-04-24T00:38:17.920724+0000","last_fullsized":"2024-04-24T00:38:17.920724+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:00:37.609101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001236025,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4505,"num_read_kb":37326,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":3953397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.19","version":"265'712","reported_seq":5857,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.778045+0000","last_change":"2024-04-24T00:38:17.778045+0000","last_active":"2024-04-24T00:38:17.778045+0000","last_peered":"2024-04-24T00:38:17.778045+0000","last_clean":"2024-04-24T00:38:17.778045+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:17.778045+0000","last_undegraded":"2024-04-24T00:38:17.778045+0000","last_fullsized":"2024-04-24T00:38:17.778045+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:04.494142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001300421,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.18","version":"263'551","reported_seq":4881,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.802371+0000","last_change":"2024-04-24T00:38:18.802601+0000","last_active":"2024-04-24T00:38:18.802371+0000","last_peered":"2024-04-24T00:38:18.802371+0000","last_clean":"2024-04-24T00:38:18.802371+0000","last_became_active":"2024-04-24T00:37:46.608136+0000","last_became_peered":"2024-04-24T00:37:46.608136+0000","last_unstale":"2024-04-24T00:38:18.802371+0000","last_undegraded":"2024-04-24T00:38:18.802371+0000","last_fullsized":"2024-04-24T00:38:18.802371+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:55:06.579921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080020700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.17","version":"263'339","reported_seq":4768,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803412+0000","last_change":"2024-04-24T00:38:18.803619+0000","last_active":"2024-04-24T00:38:18.803412+0000","last_peered":"2024-04-24T00:38:18.803412+0000","last_clean":"2024-04-24T00:38:18.803412+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:18.803412+0000","last_undegraded":"2024-04-24T00:38:18.803412+0000","last_fullsized":"2024-04-24T00:38:18.803412+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:53.921153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.16","version":"266'737","reported_seq":6716,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.785437+0000","last_change":"2024-04-24T00:38:18.756271+0000","last_active":"2024-04-24T00:38:18.785437+0000","last_peered":"2024-04-24T00:38:18.785437+0000","last_clean":"2024-04-24T00:38:18.785437+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:18.785437+0000","last_undegraded":"2024-04-24T00:38:18.785437+0000","last_fullsized":"2024-04-24T00:38:18.785437+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:36.855739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7572,"num_read_kb":50907,"num_write":1080,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.15","version":"264'632","reported_seq":7225,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.755919+0000","last_change":"2024-04-24T00:38:18.756263+0000","last_active":"2024-04-24T00:38:18.755919+0000","last_peered":"2024-04-24T00:38:18.755919+0000","last_clean":"2024-04-24T00:38:18.755919+0000","last_became_active":"2024-04-24T00:38:03.644911+0000","last_became_peered":"2024-04-24T00:38:03.644911+0000","last_unstale":"2024-04-24T00:38:18.755919+0000","last_undegraded":"2024-04-24T00:38:18.755919+0000","last_fullsized":"2024-04-24T00:38:18.755919+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:24:38.830943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.14","version":"264'496","reported_seq":5356,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.756067+0000","last_change":"2024-04-24T00:38:17.756319+0000","last_active":"2024-04-24T00:38:17.756067+0000","last_peered":"2024-04-24T00:38:17.756067+0000","last_clean":"2024-04-24T00:38:17.756067+0000","last_became_active":"2024-04-24T00:38:03.644711+0000","last_became_peered":"2024-04-24T00:38:03.644711+0000","last_unstale":"2024-04-24T00:38:17.756067+0000","last_undegraded":"2024-04-24T00:38:17.756067+0000","last_fullsized":"2024-04-24T00:38:17.756067+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:18.036694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022892910000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":508,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.754260+0000","last_change":"2024-04-24T00:35:25.695994+0000","last_active":"2024-04-24T00:38:17.754260+0000","last_peered":"2024-04-24T00:38:17.754260+0000","last_clean":"2024-04-24T00:38:17.754260+0000","last_became_active":"2024-04-24T00:35:25.695253+0000","last_became_peered":"2024-04-24T00:35:25.695253+0000","last_unstale":"2024-04-24T00:38:17.754260+0000","last_undegraded":"2024-04-24T00:38:17.754260+0000","last_fullsized":"2024-04-24T00:38:17.754260+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:15:55.543014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6478,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803861+0000","last_change":"2024-04-24T00:38:18.803861+0000","last_active":"2024-04-24T00:38:18.803861+0000","last_peered":"2024-04-24T00:38:18.803861+0000","last_clean":"2024-04-24T00:38:18.803861+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:18.803861+0000","last_undegraded":"2024-04-24T00:38:18.803861+0000","last_fullsized":"2024-04-24T00:38:18.803861+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:52.120677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":68,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.777073+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:38:17.777073+0000","last_peered":"2024-04-24T00:38:17.777073+0000","last_clean":"2024-04-24T00:38:17.777073+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:17.777073+0000","last_undegraded":"2024-04-24T00:38:17.777073+0000","last_fullsized":"2024-04-24T00:38:17.777073+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4790,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803322+0000","last_change":"2024-04-24T00:38:18.803526+0000","last_active":"2024-04-24T00:38:18.803322+0000","last_peered":"2024-04-24T00:38:18.803322+0000","last_clean":"2024-04-24T00:38:18.803322+0000","last_became_active":"2024-04-24T00:37:47.617077+0000","last_became_peered":"2024-04-24T00:37:47.617077+0000","last_unstale":"2024-04-24T00:38:18.803322+0000","last_undegraded":"2024-04-24T00:38:18.803322+0000","last_fullsized":"2024-04-24T00:38:18.803322+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:04:25.048788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001038757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.801659+0000","last_change":"2024-04-24T00:38:00.615867+0000","last_active":"2024-04-24T00:38:18.801659+0000","last_peered":"2024-04-24T00:38:18.801659+0000","last_clean":"2024-04-24T00:38:18.801659+0000","last_became_active":"2024-04-24T00:38:00.615083+0000","last_became_peered":"2024-04-24T00:38:00.615083+0000","last_unstale":"2024-04-24T00:38:18.801659+0000","last_undegraded":"2024-04-24T00:38:18.801659+0000","last_fullsized":"2024-04-24T00:38:18.801659+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:45:44.557767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4109,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.777533+0000","last_change":"2024-04-24T00:38:17.777533+0000","last_active":"2024-04-24T00:38:17.777533+0000","last_peered":"2024-04-24T00:38:17.777533+0000","last_clean":"2024-04-24T00:38:17.777533+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:17.777533+0000","last_undegraded":"2024-04-24T00:38:17.777533+0000","last_fullsized":"2024-04-24T00:38:17.777533+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:16:58.702216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077823599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":68,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.776513+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:38:17.776513+0000","last_peered":"2024-04-24T00:38:17.776513+0000","last_clean":"2024-04-24T00:38:17.776513+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:17.776513+0000","last_undegraded":"2024-04-24T00:38:17.776513+0000","last_fullsized":"2024-04-24T00:38:17.776513+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6246,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.766784+0000","last_change":"2024-04-24T00:38:18.767450+0000","last_active":"2024-04-24T00:38:18.766784+0000","last_peered":"2024-04-24T00:38:18.766784+0000","last_clean":"2024-04-24T00:38:18.766784+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:18.766784+0000","last_undegraded":"2024-04-24T00:38:18.766784+0000","last_fullsized":"2024-04-24T00:38:18.766784+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:09:34.691573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.3","version":"266'580","reported_seq":6755,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.916313+0000","last_change":"2024-04-24T00:38:18.802265+0000","last_active":"2024-04-24T00:38:18.916313+0000","last_peered":"2024-04-24T00:38:18.916313+0000","last_clean":"2024-04-24T00:38:18.916313+0000","last_became_active":"2024-04-24T00:38:04.615801+0000","last_became_peered":"2024-04-24T00:38:04.615801+0000","last_unstale":"2024-04-24T00:38:18.916313+0000","last_undegraded":"2024-04-24T00:38:18.916313+0000","last_fullsized":"2024-04-24T00:38:18.916313+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:03:27.537917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":435,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.762797+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:38:18.762797+0000","last_peered":"2024-04-24T00:38:18.762797+0000","last_clean":"2024-04-24T00:38:18.762797+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:18.762797+0000","last_undegraded":"2024-04-24T00:38:18.762797+0000","last_fullsized":"2024-04-24T00:38:18.762797+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7923,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.778133+0000","last_change":"2024-04-24T00:38:17.778390+0000","last_active":"2024-04-24T00:38:17.778133+0000","last_peered":"2024-04-24T00:38:17.778133+0000","last_clean":"2024-04-24T00:38:17.778133+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:17.778133+0000","last_undegraded":"2024-04-24T00:38:17.778133+0000","last_fullsized":"2024-04-24T00:38:17.778133+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:20.034317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014884779999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":503,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.763233+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:38:17.763233+0000","last_peered":"2024-04-24T00:38:17.763233+0000","last_clean":"2024-04-24T00:38:17.763233+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:17.763233+0000","last_undegraded":"2024-04-24T00:38:17.763233+0000","last_fullsized":"2024-04-24T00:38:17.763233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":4753,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.802150+0000","last_change":"2024-04-24T00:38:18.802387+0000","last_active":"2024-04-24T00:38:18.802150+0000","last_peered":"2024-04-24T00:38:18.802150+0000","last_clean":"2024-04-24T00:38:18.802150+0000","last_became_active":"2024-04-24T00:38:03.642971+0000","last_became_peered":"2024-04-24T00:38:03.642971+0000","last_unstale":"2024-04-24T00:38:18.802150+0000","last_undegraded":"2024-04-24T00:38:18.802150+0000","last_fullsized":"2024-04-24T00:38:18.802150+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:52.299140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.776488+0000","last_change":"2024-04-24T00:37:48.620512+0000","last_active":"2024-04-24T00:38:17.776488+0000","last_peered":"2024-04-24T00:38:17.776488+0000","last_clean":"2024-04-24T00:38:17.776488+0000","last_became_active":"2024-04-24T00:37:48.620159+0000","last_became_peered":"2024-04-24T00:37:48.620159+0000","last_unstale":"2024-04-24T00:38:17.776488+0000","last_undegraded":"2024-04-24T00:38:17.776488+0000","last_fullsized":"2024-04-24T00:38:17.776488+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:47.242898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2855,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.767788+0000","last_change":"2024-04-24T00:38:18.768267+0000","last_active":"2024-04-24T00:38:18.767788+0000","last_peered":"2024-04-24T00:38:18.767788+0000","last_clean":"2024-04-24T00:38:18.767788+0000","last_became_active":"2024-04-24T00:34:28.724174+0000","last_became_peered":"2024-04-24T00:34:28.724174+0000","last_unstale":"2024-04-24T00:38:18.767788+0000","last_undegraded":"2024-04-24T00:38:18.767788+0000","last_fullsized":"2024-04-24T00:38:18.767788+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:30:41.411539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049969319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":435,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.762994+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:38:18.762994+0000","last_peered":"2024-04-24T00:38:18.762994+0000","last_clean":"2024-04-24T00:38:18.762994+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:18.762994+0000","last_undegraded":"2024-04-24T00:38:18.762994+0000","last_fullsized":"2024-04-24T00:38:18.762994+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":463,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.801523+0000","last_change":"2024-04-24T00:37:45.610754+0000","last_active":"2024-04-24T00:38:18.801523+0000","last_peered":"2024-04-24T00:38:18.801523+0000","last_clean":"2024-04-24T00:38:18.801523+0000","last_became_active":"2024-04-24T00:37:45.609794+0000","last_became_peered":"2024-04-24T00:37:45.609794+0000","last_unstale":"2024-04-24T00:38:18.801523+0000","last_undegraded":"2024-04-24T00:38:18.801523+0000","last_fullsized":"2024-04-24T00:38:18.801523+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:27.857798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6939,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.755173+0000","last_change":"2024-04-24T00:38:18.755623+0000","last_active":"2024-04-24T00:38:18.755173+0000","last_peered":"2024-04-24T00:38:18.755173+0000","last_clean":"2024-04-24T00:38:18.755173+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:18.755173+0000","last_undegraded":"2024-04-24T00:38:18.755173+0000","last_fullsized":"2024-04-24T00:38:18.755173+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:44.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.9","version":"265'987","reported_seq":5284,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.765132+0000","last_change":"2024-04-24T00:38:17.765132+0000","last_active":"2024-04-24T00:38:17.765132+0000","last_peered":"2024-04-24T00:38:17.765132+0000","last_clean":"2024-04-24T00:38:17.765132+0000","last_became_active":"2024-04-24T00:38:04.612514+0000","last_became_peered":"2024-04-24T00:38:04.612514+0000","last_unstale":"2024-04-24T00:38:17.765132+0000","last_undegraded":"2024-04-24T00:38:17.765132+0000","last_fullsized":"2024-04-24T00:38:17.765132+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.862820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.a","version":"263'1904","reported_seq":4533,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.755501+0000","last_change":"2024-04-24T00:38:17.755920+0000","last_active":"2024-04-24T00:38:17.755501+0000","last_peered":"2024-04-24T00:38:17.755501+0000","last_clean":"2024-04-24T00:38:17.755501+0000","last_became_active":"2024-04-24T00:38:05.613032+0000","last_became_peered":"2024-04-24T00:38:05.613032+0000","last_unstale":"2024-04-24T00:38:17.755501+0000","last_undegraded":"2024-04-24T00:38:17.755501+0000","last_fullsized":"2024-04-24T00:38:17.755501+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:57:47.180836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022763219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.b","version":"263'461","reported_seq":4299,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.802498+0000","last_change":"2024-04-24T00:38:18.802777+0000","last_active":"2024-04-24T00:38:18.802498+0000","last_peered":"2024-04-24T00:38:18.802498+0000","last_clean":"2024-04-24T00:38:18.802498+0000","last_became_active":"2024-04-24T00:37:45.607329+0000","last_became_peered":"2024-04-24T00:37:45.607329+0000","last_unstale":"2024-04-24T00:38:18.802498+0000","last_undegraded":"2024-04-24T00:38:18.802498+0000","last_fullsized":"2024-04-24T00:38:18.802498+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:25:17.095603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000927724,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.c","version":"266'821","reported_seq":10059,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.766586+0000","last_change":"2024-04-24T00:38:18.766586+0000","last_active":"2024-04-24T00:38:18.766586+0000","last_peered":"2024-04-24T00:38:18.766586+0000","last_clean":"2024-04-24T00:38:18.766586+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:18.766586+0000","last_undegraded":"2024-04-24T00:38:18.766586+0000","last_fullsized":"2024-04-24T00:38:18.766586+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":821,"log_dups_size":0,"ondisk_log_size":821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:15.463491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":2786689,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10576,"num_read_kb":68776,"num_write":1168,"num_write_kb":35215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.d","version":"263'426","reported_seq":2911,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.766201+0000","last_change":"2024-04-24T00:38:18.766868+0000","last_active":"2024-04-24T00:38:18.766201+0000","last_peered":"2024-04-24T00:38:18.766201+0000","last_clean":"2024-04-24T00:38:18.766201+0000","last_became_active":"2024-04-24T00:35:28.728148+0000","last_became_peered":"2024-04-24T00:35:28.728148+0000","last_unstale":"2024-04-24T00:38:18.766201+0000","last_undegraded":"2024-04-24T00:38:18.766201+0000","last_fullsized":"2024-04-24T00:38:18.766201+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:34:24.780179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.e","version":"262'670","reported_seq":8055,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.778285+0000","last_change":"2024-04-24T00:38:17.779083+0000","last_active":"2024-04-24T00:38:17.778285+0000","last_peered":"2024-04-24T00:38:17.778285+0000","last_clean":"2024-04-24T00:38:17.778285+0000","last_became_active":"2024-04-24T00:38:03.647425+0000","last_became_peered":"2024-04-24T00:38:03.647425+0000","last_unstale":"2024-04-24T00:38:17.778285+0000","last_undegraded":"2024-04-24T00:38:17.778285+0000","last_fullsized":"2024-04-24T00:38:17.778285+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:33:42.482648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001649331,"stat_sum":{"num_bytes":3024147,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":951,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.f","version":"263'874","reported_seq":5861,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.803001+0000","last_change":"2024-04-24T00:38:18.803247+0000","last_active":"2024-04-24T00:38:18.803001+0000","last_peered":"2024-04-24T00:38:18.803001+0000","last_clean":"2024-04-24T00:38:18.803001+0000","last_became_active":"2024-04-24T00:38:04.617797+0000","last_became_peered":"2024-04-24T00:38:04.617797+0000","last_unstale":"2024-04-24T00:38:18.803001+0000","last_undegraded":"2024-04-24T00:38:18.803001+0000","last_fullsized":"2024-04-24T00:38:18.803001+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:33:23.694830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.10","version":"265'610","reported_seq":6374,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.885339+0000","last_change":"2024-04-24T00:38:17.755366+0000","last_active":"2024-04-24T00:38:17.885339+0000","last_peered":"2024-04-24T00:38:17.885339+0000","last_clean":"2024-04-24T00:38:17.885339+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:17.885339+0000","last_undegraded":"2024-04-24T00:38:17.885339+0000","last_fullsized":"2024-04-24T00:38:17.885339+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:16:28.900971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001829174,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.11","version":"265'598","reported_seq":8447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.869475+0000","last_change":"2024-04-24T00:38:17.755488+0000","last_active":"2024-04-24T00:38:17.869475+0000","last_peered":"2024-04-24T00:38:17.869475+0000","last_clean":"2024-04-24T00:38:17.869475+0000","last_became_active":"2024-04-24T00:38:04.614076+0000","last_became_peered":"2024-04-24T00:38:04.614076+0000","last_unstale":"2024-04-24T00:38:17.869475+0000","last_undegraded":"2024-04-24T00:38:17.869475+0000","last_fullsized":"2024-04-24T00:38:17.869475+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:19:50.707765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.12","version":"265'1515","reported_seq":6097,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-24T00:38:18.801822+0000","last_change":"2024-04-24T00:38:18.802080+0000","last_active":"2024-04-24T00:38:18.801822+0000","last_peered":"2024-04-24T00:38:18.801822+0000","last_clean":"2024-04-24T00:38:18.801822+0000","last_became_active":"2024-04-24T00:37:52.611238+0000","last_became_peered":"2024-04-24T00:37:52.611238+0000","last_unstale":"2024-04-24T00:38:18.801822+0000","last_undegraded":"2024-04-24T00:38:18.801822+0000","last_fullsized":"2024-04-24T00:38:18.801822+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:46:09.244438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]},{"pgid":"3.13","version":"263'651","reported_seq":5048,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-24T00:38:17.763982+0000","last_change":"2024-04-24T00:38:17.764335+0000","last_active":"2024-04-24T00:38:17.763982+0000","last_peered":"2024-04-24T00:38:17.763982+0000","last_clean":"2024-04-24T00:38:17.763982+0000","last_became_active":"2024-04-24T00:38:04.612883+0000","last_became_peered":"2024-04-24T00:38:04.612883+0000","last_unstale":"2024-04-24T00:38:17.763982+0000","last_undegraded":"2024-04-24T00:38:17.763982+0000","last_fullsized":"2024-04-24T00:38:17.763982+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:40:45.891223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001199851,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"},{"start":"9","length":"1"},{"start":"b","length":"1"},{"start":"d","length":"1"},{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"1"},{"start":"17","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5810855,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":185533,"num_read_kb":1545457,"num_write":31193,"num_write_kb":1018169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6894,"num_bytes_recovered":358075051,"num_keys_recovered":554,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5619712,"data_stored":36462251,"data_compressed":1807200,"data_compressed_allocated":4071424,"data_compressed_original":35677204,"omap_allocated":4834,"internal_metadata":0},"log_size":22799,"ondisk_log_size":22799,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2951840,"data_compressed":34064,"data_compressed_allocated":184320,"data_compressed_original":2932736,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574909,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66296,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305544,"statfs":{"total":96636764160,"available":96568877056,"internally_reserved":0,"allocated":3256320,"data_stored":2191242,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70440,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301400,"statfs":{"total":96636764160,"available":96564633600,"internally_reserved":0,"allocated":3235840,"data_stored":1626319,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574908,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74596,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297244,"statfs":{"total":96636764160,"available":96560377856,"internally_reserved":0,"allocated":3231744,"data_stored":1626303,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574909,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60812,"kb_used_data":3264,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311028,"statfs":{"total":96636764160,"available":96574492672,"internally_reserved":0,"allocated":3342336,"data_stored":1747857,"data_compressed":89589,"data_compressed_allocated":94208,"data_compressed_original":135168,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60784,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311056,"statfs":{"total":96636764160,"available":96574521344,"internally_reserved":0,"allocated":3231744,"data_stored":1636410,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893158,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35088,"kb_used_data":3192,"kb_used_omap":6,"kb_used_meta":31865,"kb_avail":94336752,"statfs":{"total":96636764160,"available":96600834048,"internally_reserved":0,"allocated":3268608,"data_stored":2197695,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":32630499},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6124,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":35536,"data_compressed":63,"data_compressed_allocated":4096,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":25429,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1863680,"data_stored":11425356,"data_compressed":520515,"data_compressed_allocated":1241088,"data_compressed_original":11121490,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":25445,"data_compressed":54,"data_compressed_allocated":4096,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3698688,"data_stored":24950466,"data_compressed":1286514,"data_compressed_allocated":2818048,"data_compressed_original":24481986,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:38:22.581 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T00:38:22.581 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T00:38:22.581 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-24T00:38:22.582 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T00:38:22.582 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T00:38:22.582 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-24T00:38:22.711 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-04-24T00:38:22.712 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "" 2024-04-24T00:38:22.712 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-04-24T00:38:22.810 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-04-24T00:38:27.726 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-24T00:38:27.727 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-04-24T00:38:27.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-04-24T00:38:28.049 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:38:28.046+0000 7f841a5f8640 -1 mon.a@0(leader).osd e271 definitely_dead 0 2024-04-24T00:38:28.823 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.4. 2024-04-24T00:38:28.853 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-04-24T00:38:28.853 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-24T00:38:28.853 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-04-24T00:38:28.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-04-24T00:38:29.167 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-04-24T00:38:29.162+0000 7f841a5f8640 -1 mon.a@0(leader).osd e272 definitely_dead 0 2024-04-24T00:38:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.7. 2024-04-24T00:38:29.644 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-04-24T00:38:29.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-24T00:38:29.785 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a: {} 2024-04-24T00:38:29.785 INFO:teuthology.orchestra.run.smithi064.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b: {} 2024-04-24T00:38:29.799 INFO:teuthology.orchestra.run.smithi064.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T00:38:29.809 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c: {} 2024-04-24T00:38:29.809 INFO:teuthology.orchestra.run.smithi064.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T00:38:29.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T00:38:30.618 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T00:38:30.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-24T00:38:31.620 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-24T00:38:31.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T00:38:32.629 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T00:38:32.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-24T00:38:33.631 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-24T00:38:33.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T00:38:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T00:38:34.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-24T00:38:35.613 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-24T00:38:35.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T00:38:36.623 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T00:38:36.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-24T00:38:37.625 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-24T00:38:37.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T00:38:38.628 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T00:38:38.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-24T00:38:39.854 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-24T00:38:39.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-24T00:38:40.857 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-04-24T00:38:40.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-24T00:38:41.860 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-24T00:38:41.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-24T00:38:42.870 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.7 to 1 (10000) 2024-04-24T00:38:42.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-24T00:38:43.898 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-24T00:38:43.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T00:38:44.895 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T00:38:44.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-24T00:38:45.911 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-24T00:38:45.938 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-24T00:38:45.938 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-24T00:38:45.938 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-24T00:38:45.938 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-24T00:38:45.938 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T00:38:45.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T00:38:46.245 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:46.245 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":289,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:38:45.901888+0000","last_up_change":"2024-04-24T00:35:14.574643+0000","last_in_change":"2024-04-24T00:38:29.604271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:33:39.734814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:33:43.786311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-04-24T00:33:51.288702+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":85,"snap_epoch":266,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":74,"up_from":95,"up_thru":281,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6817","nonce":218259947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6819","nonce":218259947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6823","nonce":218259947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6821","nonce":218259947}]},"public_addr":"172.21.15.64:6817/218259947","cluster_addr":"172.21.15.64:6819/218259947","heartbeat_back_addr":"172.21.15.64:6823/218259947","heartbeat_front_addr":"172.21.15.64:6821/218259947","state":["exists","up"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6825","nonce":637720411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6827","nonce":637720411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6831","nonce":637720411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6829","nonce":637720411}]},"public_addr":"172.21.15.64:6825/637720411","cluster_addr":"172.21.15.64:6827/637720411","heartbeat_back_addr":"172.21.15.64:6831/637720411","heartbeat_front_addr":"172.21.15.64:6829/637720411","state":["exists","up"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6809","nonce":820024667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6811","nonce":820024667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6815","nonce":820024667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6813","nonce":820024667}]},"public_addr":"172.21.15.64:6809/820024667","cluster_addr":"172.21.15.64:6811/820024667","heartbeat_back_addr":"172.21.15.64:6815/820024667","heartbeat_front_addr":"172.21.15.64:6813/820024667","state":["exists","up"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":279,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6801","nonce":1974534323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6803","nonce":1974534323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6807","nonce":1974534323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6805","nonce":1974534323}]},"public_addr":"172.21.15.64:6801/1974534323","cluster_addr":"172.21.15.64:6803/1974534323","heartbeat_back_addr":"172.21.15.64:6807/1974534323","heartbeat_front_addr":"172.21.15.64:6805/1974534323","state":["exists","up"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":278,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6818","nonce":1642543604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6821","nonce":1642543604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6828","nonce":1642543604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6823","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6824","nonce":1642543604}]},"public_addr":"172.21.15.167:6818/1642543604","cluster_addr":"172.21.15.167:6821/1642543604","heartbeat_back_addr":"172.21.15.167:6828/1642543604","heartbeat_front_addr":"172.21.15.167:6824/1642543604","state":["exists","up"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":278,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6809","nonce":172726194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6811","nonce":172726194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6815","nonce":172726194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6813","nonce":172726194}]},"public_addr":"172.21.15.167:6809/172726194","cluster_addr":"172.21.15.167:6811/172726194","heartbeat_back_addr":"172.21.15.167:6815/172726194","heartbeat_front_addr":"172.21.15.167:6813/172726194","state":["exists","up"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6801","nonce":2195746987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6803","nonce":2195746987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6807","nonce":2195746987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6805","nonce":2195746987}]},"public_addr":"172.21.15.167:6801/2195746987","cluster_addr":"172.21.15.167:6803/2195746987","heartbeat_back_addr":"172.21.15.167:6807/2195746987","heartbeat_front_addr":"172.21.15.167:6805/2195746987","state":["exists","up"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":280,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6817","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6819","nonce":954791674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6825","nonce":954791674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6831","nonce":954791674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6827","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6829","nonce":954791674}]},"public_addr":"172.21.15.167:6819/954791674","cluster_addr":"172.21.15.167:6825/954791674","heartbeat_back_addr":"172.21.15.167:6831/954791674","heartbeat_front_addr":"172.21.15.167:6829/954791674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.509233+0000","dead_epoch":75},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.465696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.405177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.269202+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.359180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.351834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.490127+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.484842+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.16","osds":[3,2]},{"pgid":"3.1b","osds":[5,6]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:38:46.264 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T00:38:46.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T00:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":289,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:38:45.901888+0000","last_up_change":"2024-04-24T00:35:14.574643+0000","last_in_change":"2024-04-24T00:38:29.604271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:33:39.734814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:33:43.786311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-04-24T00:33:51.288702+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":85,"snap_epoch":266,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":74,"up_from":95,"up_thru":281,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6817","nonce":218259947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6819","nonce":218259947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6823","nonce":218259947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6821","nonce":218259947}]},"public_addr":"172.21.15.64:6817/218259947","cluster_addr":"172.21.15.64:6819/218259947","heartbeat_back_addr":"172.21.15.64:6823/218259947","heartbeat_front_addr":"172.21.15.64:6821/218259947","state":["exists","up"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6825","nonce":637720411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6827","nonce":637720411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6831","nonce":637720411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6829","nonce":637720411}]},"public_addr":"172.21.15.64:6825/637720411","cluster_addr":"172.21.15.64:6827/637720411","heartbeat_back_addr":"172.21.15.64:6831/637720411","heartbeat_front_addr":"172.21.15.64:6829/637720411","state":["exists","up"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6809","nonce":820024667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6811","nonce":820024667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6815","nonce":820024667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6813","nonce":820024667}]},"public_addr":"172.21.15.64:6809/820024667","cluster_addr":"172.21.15.64:6811/820024667","heartbeat_back_addr":"172.21.15.64:6815/820024667","heartbeat_front_addr":"172.21.15.64:6813/820024667","state":["exists","up"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":279,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6801","nonce":1974534323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6803","nonce":1974534323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6807","nonce":1974534323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6805","nonce":1974534323}]},"public_addr":"172.21.15.64:6801/1974534323","cluster_addr":"172.21.15.64:6803/1974534323","heartbeat_back_addr":"172.21.15.64:6807/1974534323","heartbeat_front_addr":"172.21.15.64:6805/1974534323","state":["exists","up"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":278,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6818","nonce":1642543604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6821","nonce":1642543604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6828","nonce":1642543604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6823","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6824","nonce":1642543604}]},"public_addr":"172.21.15.167:6818/1642543604","cluster_addr":"172.21.15.167:6821/1642543604","heartbeat_back_addr":"172.21.15.167:6828/1642543604","heartbeat_front_addr":"172.21.15.167:6824/1642543604","state":["exists","up"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":278,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6809","nonce":172726194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6811","nonce":172726194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6815","nonce":172726194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6813","nonce":172726194}]},"public_addr":"172.21.15.167:6809/172726194","cluster_addr":"172.21.15.167:6811/172726194","heartbeat_back_addr":"172.21.15.167:6815/172726194","heartbeat_front_addr":"172.21.15.167:6813/172726194","state":["exists","up"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6801","nonce":2195746987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6803","nonce":2195746987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6807","nonce":2195746987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6805","nonce":2195746987}]},"public_addr":"172.21.15.167:6801/2195746987","cluster_addr":"172.21.15.167:6803/2195746987","heartbeat_back_addr":"172.21.15.167:6807/2195746987","heartbeat_front_addr":"172.21.15.167:6805/2195746987","state":["exists","up"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":280,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6817","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6819","nonce":954791674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6825","nonce":954791674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6831","nonce":954791674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6827","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6829","nonce":954791674}]},"public_addr":"172.21.15.167:6819/954791674","cluster_addr":"172.21.15.167:6825/954791674","heartbeat_back_addr":"172.21.15.167:6831/954791674","heartbeat_front_addr":"172.21.15.167:6829/954791674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.509233+0000","dead_epoch":75},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.465696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.405177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.269202+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.359180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.351834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.490127+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.484842+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.16","osds":[3,2]},{"pgid":"3.1b","osds":[5,6]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:38:46.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T00:38:46.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T00:38:46.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T00:38:46.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T00:38:46.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T00:38:46.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T00:38:46.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-24T00:38:46.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-24T00:38:46.837 INFO:teuthology.orchestra.run.smithi064.stdout:408021893164 2024-04-24T00:38:46.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T00:38:46.845 INFO:teuthology.orchestra.run.smithi064.stdout:55834574915 2024-04-24T00:38:46.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T00:38:46.858 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:46.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T00:38:46.864 INFO:teuthology.orchestra.run.smithi064.stdout:55834574915 2024-04-24T00:38:46.864 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:46.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T00:38:46.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T00:38:46.867 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:46.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T00:38:46.869 INFO:teuthology.orchestra.run.smithi064.stdout:55834574913 2024-04-24T00:38:46.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T00:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout:55834574915 2024-04-24T00:38:47.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T00:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout:408021893163 2024-04-24T00:38:47.316 INFO:tasks.ceph.ceph_manager.ceph:need seq 408021893164 got 408021893163 for osd.0 2024-04-24T00:38:47.346 INFO:teuthology.orchestra.run.smithi064.stdout:55834574913 2024-04-24T00:38:47.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574914 got 55834574913 for osd.6 2024-04-24T00:38:47.366 INFO:teuthology.orchestra.run.smithi064.stdout:55834574913 2024-04-24T00:38:47.380 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:47.387 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574914 got 55834574913 for osd.4 2024-04-24T00:38:47.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574915 got 55834574914 for osd.3 2024-04-24T00:38:47.413 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:47.429 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574915 got 55834574914 for osd.2 2024-04-24T00:38:47.462 INFO:teuthology.orchestra.run.smithi064.stdout:55834574913 2024-04-24T00:38:47.477 INFO:teuthology.orchestra.run.smithi064.stdout:55834574912 2024-04-24T00:38:47.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574914 got 55834574913 for osd.1 2024-04-24T00:38:47.493 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574913 got 55834574912 for osd.5 2024-04-24T00:38:47.503 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:47.520 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574915 got 55834574914 for osd.7 2024-04-24T00:38:48.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T00:38:48.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T00:38:48.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T00:38:48.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T00:38:48.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T00:38:48.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T00:38:48.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T00:38:48.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T00:38:48.725 INFO:teuthology.orchestra.run.smithi064.stdout:408021893164 2024-04-24T00:38:48.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 408021893164 got 408021893164 for osd.0 2024-04-24T00:38:48.751 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:48.852 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:48.879 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574914 got 55834574914 for osd.6 2024-04-24T00:38:48.879 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:48.901 INFO:teuthology.orchestra.run.smithi064.stdout:55834574915 2024-04-24T00:38:48.912 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:48.927 INFO:teuthology.orchestra.run.smithi064.stdout:55834574914 2024-04-24T00:38:48.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574915 got 55834574915 for osd.3 2024-04-24T00:38:48.928 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:48.937 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574914 got 55834574914 for osd.1 2024-04-24T00:38:48.937 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:48.946 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574914 got 55834574914 for osd.4 2024-04-24T00:38:48.946 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:48.949 INFO:teuthology.orchestra.run.smithi064.stdout:55834574913 2024-04-24T00:38:48.966 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574913 got 55834574913 for osd.5 2024-04-24T00:38:48.966 INFO:teuthology.orchestra.run.smithi064.stdout:55834574915 2024-04-24T00:38:48.967 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:48.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574915 got 55834574915 for osd.2 2024-04-24T00:38:48.984 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:49.031 INFO:teuthology.orchestra.run.smithi064.stdout:55834574915 2024-04-24T00:38:49.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574915 got 55834574915 for osd.7 2024-04-24T00:38:49.049 DEBUG:teuthology.parallel:result is None 2024-04-24T00:38:49.049 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T00:38:49.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:38:49.383 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:49.384 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:38:49.400 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-04-24T00:38:47.729265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":185758,"num_read_kb":1545653,"num_write":31374,"num_write_kb":1020589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8595,"num_bytes_recovered":360436542,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22942,"ondisk_log_size":22942,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":527496,"kb_used_data":27464,"kb_used_omap":17,"kb_used_meta":499886,"kb_avail":754447224,"statfs":{"total":773094113280,"available":772553957376,"internally_reserved":0,"allocated":28123136,"data_stored":15132594,"data_compressed":13970,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":17586,"internal_metadata":511884110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.857035"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":5999,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062969+0000","last_change":"2024-04-24T00:38:18.803340+0000","last_active":"2024-04-24T00:38:46.062969+0000","last_peered":"2024-04-24T00:38:46.062969+0000","last_clean":"2024-04-24T00:38:46.062969+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:46.062969+0000","last_undegraded":"2024-04-24T00:38:46.062969+0000","last_fullsized":"2024-04-24T00:38:46.062969+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:16.140487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"264'1050","reported_seq":5683,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370225+0000","last_change":"2024-04-24T00:38:36.641262+0000","last_active":"2024-04-24T00:38:46.370225+0000","last_peered":"2024-04-24T00:38:46.370225+0000","last_clean":"2024-04-24T00:38:46.370225+0000","last_became_active":"2024-04-24T00:38:36.639573+0000","last_became_peered":"2024-04-24T00:38:36.639573+0000","last_unstale":"2024-04-24T00:38:46.370225+0000","last_undegraded":"2024-04-24T00:38:46.370225+0000","last_fullsized":"2024-04-24T00:38:46.370225+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:18:26.284735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080331599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"263'591","reported_seq":7547,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911549+0000","last_change":"2024-04-24T00:38:33.652375+0000","last_active":"2024-04-24T00:38:45.911549+0000","last_peered":"2024-04-24T00:38:45.911549+0000","last_clean":"2024-04-24T00:38:45.911549+0000","last_became_active":"2024-04-24T00:38:33.649639+0000","last_became_peered":"2024-04-24T00:38:33.649639+0000","last_unstale":"2024-04-24T00:38:45.911549+0000","last_undegraded":"2024-04-24T00:38:45.911549+0000","last_fullsized":"2024-04-24T00:38:45.911549+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:36.602084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"266'795","reported_seq":6337,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:38:35.634698+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:38:35.628937+0000","last_became_peered":"2024-04-24T00:38:35.628937+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:48.395425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5266,"num_read_kb":61368,"num_write":1008,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"264'498","reported_seq":4556,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911438+0000","last_change":"2024-04-24T00:38:18.764647+0000","last_active":"2024-04-24T00:38:45.911438+0000","last_peered":"2024-04-24T00:38:45.911438+0000","last_clean":"2024-04-24T00:38:45.911438+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:45.911438+0000","last_undegraded":"2024-04-24T00:38:45.911438+0000","last_fullsized":"2024-04-24T00:38:45.911438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:59.132716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"265'390","reported_seq":6310,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910619+0000","last_change":"2024-04-24T00:38:35.638402+0000","last_active":"2024-04-24T00:38:45.910619+0000","last_peered":"2024-04-24T00:38:45.910619+0000","last_clean":"2024-04-24T00:38:45.910619+0000","last_became_active":"2024-04-24T00:38:35.631522+0000","last_became_peered":"2024-04-24T00:38:35.631522+0000","last_unstale":"2024-04-24T00:38:45.910619+0000","last_undegraded":"2024-04-24T00:38:45.910619+0000","last_fullsized":"2024-04-24T00:38:45.910619+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:44.645527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001287217,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4509,"num_read_kb":37330,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":3953416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"265'712","reported_seq":5909,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909836+0000","last_change":"2024-04-24T00:38:18.779600+0000","last_active":"2024-04-24T00:38:45.909836+0000","last_peered":"2024-04-24T00:38:45.909836+0000","last_clean":"2024-04-24T00:38:45.909836+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:45.909836+0000","last_undegraded":"2024-04-24T00:38:45.909836+0000","last_fullsized":"2024-04-24T00:38:45.909836+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:58.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059221100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"263'551","reported_seq":4877,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.369675+0000","last_change":"2024-04-24T00:38:36.632405+0000","last_active":"2024-04-24T00:38:46.369675+0000","last_peered":"2024-04-24T00:38:46.369675+0000","last_clean":"2024-04-24T00:38:46.369675+0000","last_became_active":"2024-04-24T00:38:36.631395+0000","last_became_peered":"2024-04-24T00:38:36.631395+0000","last_unstale":"2024-04-24T00:38:46.369675+0000","last_undegraded":"2024-04-24T00:38:46.369675+0000","last_fullsized":"2024-04-24T00:38:46.369675+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:35:30.536918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001058769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"263'339","reported_seq":4813,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063021+0000","last_change":"2024-04-24T00:38:18.803619+0000","last_active":"2024-04-24T00:38:46.063021+0000","last_peered":"2024-04-24T00:38:46.063021+0000","last_clean":"2024-04-24T00:38:46.063021+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:46.063021+0000","last_undegraded":"2024-04-24T00:38:46.063021+0000","last_fullsized":"2024-04-24T00:38:46.063021+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:53.921153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"266'751","reported_seq":6814,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370092+0000","last_change":"2024-04-24T00:38:18.756271+0000","last_active":"2024-04-24T00:38:46.370092+0000","last_peered":"2024-04-24T00:38:46.370092+0000","last_clean":"2024-04-24T00:38:46.370092+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:46.370092+0000","last_undegraded":"2024-04-24T00:38:46.370092+0000","last_fullsized":"2024-04-24T00:38:46.370092+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:36.855739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7650,"num_read_kb":50977,"num_write":1087,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"264'632","reported_seq":7275,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813710+0000","last_change":"2024-04-24T00:38:32.649992+0000","last_active":"2024-04-24T00:38:46.813710+0000","last_peered":"2024-04-24T00:38:46.813710+0000","last_clean":"2024-04-24T00:38:46.813710+0000","last_became_active":"2024-04-24T00:38:32.643949+0000","last_became_peered":"2024-04-24T00:38:32.643949+0000","last_unstale":"2024-04-24T00:38:46.813710+0000","last_undegraded":"2024-04-24T00:38:46.813710+0000","last_fullsized":"2024-04-24T00:38:46.813710+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:29:36.206394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"264'496","reported_seq":5383,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911030+0000","last_change":"2024-04-24T00:38:34.636361+0000","last_active":"2024-04-24T00:38:45.911030+0000","last_peered":"2024-04-24T00:38:45.911030+0000","last_clean":"2024-04-24T00:38:45.911030+0000","last_became_active":"2024-04-24T00:38:34.632609+0000","last_became_peered":"2024-04-24T00:38:34.632609+0000","last_unstale":"2024-04-24T00:38:45.911030+0000","last_undegraded":"2024-04-24T00:38:45.911030+0000","last_fullsized":"2024-04-24T00:38:45.911030+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:51:37.284131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010187200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":560,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107558+0000","last_change":"2024-04-24T00:38:30.696852+0000","last_active":"2024-04-24T00:38:46.107558+0000","last_peered":"2024-04-24T00:38:46.107558+0000","last_clean":"2024-04-24T00:38:46.107558+0000","last_became_active":"2024-04-24T00:38:30.696223+0000","last_became_peered":"2024-04-24T00:38:30.696223+0000","last_unstale":"2024-04-24T00:38:46.107558+0000","last_undegraded":"2024-04-24T00:38:46.107558+0000","last_fullsized":"2024-04-24T00:38:46.107558+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:08:35.422980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6523,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063089+0000","last_change":"2024-04-24T00:38:18.803861+0000","last_active":"2024-04-24T00:38:46.063089+0000","last_peered":"2024-04-24T00:38:46.063089+0000","last_clean":"2024-04-24T00:38:46.063089+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:46.063089+0000","last_undegraded":"2024-04-24T00:38:46.063089+0000","last_fullsized":"2024-04-24T00:38:46.063089+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:52.120677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910495+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:38:45.910495+0000","last_peered":"2024-04-24T00:38:45.910495+0000","last_clean":"2024-04-24T00:38:45.910495+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:45.910495+0000","last_undegraded":"2024-04-24T00:38:45.910495+0000","last_fullsized":"2024-04-24T00:38:45.910495+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4786,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370362+0000","last_change":"2024-04-24T00:38:36.639909+0000","last_active":"2024-04-24T00:38:46.370362+0000","last_peered":"2024-04-24T00:38:46.370362+0000","last_clean":"2024-04-24T00:38:46.370362+0000","last_became_active":"2024-04-24T00:38:36.637958+0000","last_became_peered":"2024-04-24T00:38:36.637958+0000","last_unstale":"2024-04-24T00:38:46.370362+0000","last_undegraded":"2024-04-24T00:38:46.370362+0000","last_fullsized":"2024-04-24T00:38:46.370362+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:23.662984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017453659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813411+0000","last_change":"2024-04-24T00:38:30.641161+0000","last_active":"2024-04-24T00:38:46.813411+0000","last_peered":"2024-04-24T00:38:46.813411+0000","last_clean":"2024-04-24T00:38:46.813411+0000","last_became_active":"2024-04-24T00:38:30.640288+0000","last_became_peered":"2024-04-24T00:38:30.640288+0000","last_unstale":"2024-04-24T00:38:46.813411+0000","last_undegraded":"2024-04-24T00:38:46.813411+0000","last_fullsized":"2024-04-24T00:38:46.813411+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:46:24.491722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4161,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910063+0000","last_change":"2024-04-24T00:38:18.780633+0000","last_active":"2024-04-24T00:38:45.910063+0000","last_peered":"2024-04-24T00:38:45.910063+0000","last_clean":"2024-04-24T00:38:45.910063+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:45.910063+0000","last_undegraded":"2024-04-24T00:38:45.910063+0000","last_fullsized":"2024-04-24T00:38:45.910063+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:46.783523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099788599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909769+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:38:45.909769+0000","last_peered":"2024-04-24T00:38:45.909769+0000","last_clean":"2024-04-24T00:38:45.909769+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:45.909769+0000","last_undegraded":"2024-04-24T00:38:45.909769+0000","last_fullsized":"2024-04-24T00:38:45.909769+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6292,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911342+0000","last_change":"2024-04-24T00:38:18.767450+0000","last_active":"2024-04-24T00:38:45.911342+0000","last_peered":"2024-04-24T00:38:45.911342+0000","last_clean":"2024-04-24T00:38:45.911342+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:45.911342+0000","last_undegraded":"2024-04-24T00:38:45.911342+0000","last_fullsized":"2024-04-24T00:38:45.911342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:09:34.691573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"266'580","reported_seq":6804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910527+0000","last_change":"2024-04-24T00:38:32.649496+0000","last_active":"2024-04-24T00:38:45.910527+0000","last_peered":"2024-04-24T00:38:45.910527+0000","last_clean":"2024-04-24T00:38:45.910527+0000","last_became_active":"2024-04-24T00:38:32.645508+0000","last_became_peered":"2024-04-24T00:38:32.645508+0000","last_unstale":"2024-04-24T00:38:45.910527+0000","last_undegraded":"2024-04-24T00:38:45.910527+0000","last_fullsized":"2024-04-24T00:38:45.910527+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:59.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910983+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:38:45.910983+0000","last_peered":"2024-04-24T00:38:45.910983+0000","last_clean":"2024-04-24T00:38:45.910983+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:45.910983+0000","last_undegraded":"2024-04-24T00:38:45.910983+0000","last_fullsized":"2024-04-24T00:38:45.910983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7975,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909664+0000","last_change":"2024-04-24T00:38:18.780209+0000","last_active":"2024-04-24T00:38:45.909664+0000","last_peered":"2024-04-24T00:38:45.909664+0000","last_clean":"2024-04-24T00:38:45.909664+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:45.909664+0000","last_undegraded":"2024-04-24T00:38:45.909664+0000","last_fullsized":"2024-04-24T00:38:45.909664+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:49.181426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":550,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370389+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:38:46.370389+0000","last_peered":"2024-04-24T00:38:46.370389+0000","last_clean":"2024-04-24T00:38:46.370389+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:46.370389+0000","last_undegraded":"2024-04-24T00:38:46.370389+0000","last_fullsized":"2024-04-24T00:38:46.370389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":5035,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063003+0000","last_change":"2024-04-24T00:38:34.632199+0000","last_active":"2024-04-24T00:38:46.063003+0000","last_peered":"2024-04-24T00:38:46.063003+0000","last_clean":"2024-04-24T00:38:46.063003+0000","last_became_active":"2024-04-24T00:38:34.629456+0000","last_became_peered":"2024-04-24T00:38:34.629456+0000","last_unstale":"2024-04-24T00:38:46.063003+0000","last_undegraded":"2024-04-24T00:38:46.063003+0000","last_fullsized":"2024-04-24T00:38:46.063003+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:05:27.920476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813416+0000","last_change":"2024-04-24T00:38:30.634583+0000","last_active":"2024-04-24T00:38:46.813416+0000","last_peered":"2024-04-24T00:38:46.813416+0000","last_clean":"2024-04-24T00:38:46.813416+0000","last_became_active":"2024-04-24T00:38:30.633606+0000","last_became_peered":"2024-04-24T00:38:30.633606+0000","last_unstale":"2024-04-24T00:38:46.813416+0000","last_undegraded":"2024-04-24T00:38:46.813416+0000","last_fullsized":"2024-04-24T00:38:46.813416+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:24:57.510774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2873,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.369748+0000","last_change":"2024-04-24T00:38:36.633952+0000","last_active":"2024-04-24T00:38:46.369748+0000","last_peered":"2024-04-24T00:38:46.369748+0000","last_clean":"2024-04-24T00:38:46.369748+0000","last_became_active":"2024-04-24T00:38:36.632239+0000","last_became_peered":"2024-04-24T00:38:36.632239+0000","last_unstale":"2024-04-24T00:38:46.369748+0000","last_undegraded":"2024-04-24T00:38:46.369748+0000","last_fullsized":"2024-04-24T00:38:46.369748+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:50.888554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012896030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":506,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813566+0000","last_change":"2024-04-24T00:38:36.642364+0000","last_active":"2024-04-24T00:38:46.813566+0000","last_peered":"2024-04-24T00:38:46.813566+0000","last_clean":"2024-04-24T00:38:46.813566+0000","last_became_active":"2024-04-24T00:38:36.641100+0000","last_became_peered":"2024-04-24T00:38:36.641100+0000","last_unstale":"2024-04-24T00:38:46.813566+0000","last_undegraded":"2024-04-24T00:38:46.813566+0000","last_fullsized":"2024-04-24T00:38:46.813566+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:00.586831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6983,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.369772+0000","last_change":"2024-04-24T00:38:18.755623+0000","last_active":"2024-04-24T00:38:46.369772+0000","last_peered":"2024-04-24T00:38:46.369772+0000","last_clean":"2024-04-24T00:38:46.369772+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:46.369772+0000","last_undegraded":"2024-04-24T00:38:46.369772+0000","last_fullsized":"2024-04-24T00:38:46.369772+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:44.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"265'987","reported_seq":5319,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910553+0000","last_change":"2024-04-24T00:38:33.656528+0000","last_active":"2024-04-24T00:38:45.910553+0000","last_peered":"2024-04-24T00:38:45.910553+0000","last_clean":"2024-04-24T00:38:45.910553+0000","last_became_active":"2024-04-24T00:38:33.650599+0000","last_became_peered":"2024-04-24T00:38:33.650599+0000","last_unstale":"2024-04-24T00:38:45.910553+0000","last_undegraded":"2024-04-24T00:38:45.910553+0000","last_fullsized":"2024-04-24T00:38:45.910553+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:19.834939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"263'1904","reported_seq":4804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107528+0000","last_change":"2024-04-24T00:38:34.638496+0000","last_active":"2024-04-24T00:38:46.107528+0000","last_peered":"2024-04-24T00:38:46.107528+0000","last_clean":"2024-04-24T00:38:46.107528+0000","last_became_active":"2024-04-24T00:38:34.634199+0000","last_became_peered":"2024-04-24T00:38:34.634199+0000","last_unstale":"2024-04-24T00:38:46.107528+0000","last_undegraded":"2024-04-24T00:38:46.107528+0000","last_fullsized":"2024-04-24T00:38:46.107528+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:57.455108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001715414,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"263'461","reported_seq":4301,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.369928+0000","last_change":"2024-04-24T00:38:36.634741+0000","last_active":"2024-04-24T00:38:46.369928+0000","last_peered":"2024-04-24T00:38:46.369928+0000","last_clean":"2024-04-24T00:38:46.369928+0000","last_became_active":"2024-04-24T00:38:36.633468+0000","last_became_peered":"2024-04-24T00:38:36.633468+0000","last_unstale":"2024-04-24T00:38:46.369928+0000","last_undegraded":"2024-04-24T00:38:46.369928+0000","last_fullsized":"2024-04-24T00:38:46.369928+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:40.363226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002035177,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"266'831","reported_seq":10117,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911887+0000","last_change":"2024-04-24T00:38:18.766586+0000","last_active":"2024-04-24T00:38:45.911887+0000","last_peered":"2024-04-24T00:38:45.911887+0000","last_clean":"2024-04-24T00:38:45.911887+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:45.911887+0000","last_undegraded":"2024-04-24T00:38:45.911887+0000","last_fullsized":"2024-04-24T00:38:45.911887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:15.463491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10600,"num_read_kb":68795,"num_write":1180,"num_write_kb":35219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"263'426","reported_seq":2956,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813960+0000","last_change":"2024-04-24T00:38:35.637609+0000","last_active":"2024-04-24T00:38:46.813960+0000","last_peered":"2024-04-24T00:38:46.813960+0000","last_clean":"2024-04-24T00:38:46.813960+0000","last_became_active":"2024-04-24T00:38:35.632277+0000","last_became_peered":"2024-04-24T00:38:35.632277+0000","last_unstale":"2024-04-24T00:38:46.813960+0000","last_undegraded":"2024-04-24T00:38:46.813960+0000","last_fullsized":"2024-04-24T00:38:46.813960+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:03.305609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"266'671","reported_seq":8093,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814115+0000","last_change":"2024-04-24T00:38:37.642821+0000","last_active":"2024-04-24T00:38:46.814115+0000","last_peered":"2024-04-24T00:38:46.814115+0000","last_clean":"2024-04-24T00:38:46.814115+0000","last_became_active":"2024-04-24T00:38:37.638998+0000","last_became_peered":"2024-04-24T00:38:37.638998+0000","last_unstale":"2024-04-24T00:38:46.814115+0000","last_undegraded":"2024-04-24T00:38:46.814115+0000","last_fullsized":"2024-04-24T00:38:46.814115+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:10:52.647284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001218907,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":952,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"263'874","reported_seq":5902,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814247+0000","last_change":"2024-04-24T00:38:36.646741+0000","last_active":"2024-04-24T00:38:46.814247+0000","last_peered":"2024-04-24T00:38:46.814247+0000","last_clean":"2024-04-24T00:38:46.814247+0000","last_became_active":"2024-04-24T00:38:36.640462+0000","last_became_peered":"2024-04-24T00:38:36.640462+0000","last_unstale":"2024-04-24T00:38:46.814247+0000","last_undegraded":"2024-04-24T00:38:46.814247+0000","last_fullsized":"2024-04-24T00:38:46.814247+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:25.891296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"265'610","reported_seq":6427,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107457+0000","last_change":"2024-04-24T00:38:18.844983+0000","last_active":"2024-04-24T00:38:46.107457+0000","last_peered":"2024-04-24T00:38:46.107457+0000","last_clean":"2024-04-24T00:38:46.107457+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:46.107457+0000","last_undegraded":"2024-04-24T00:38:46.107457+0000","last_fullsized":"2024-04-24T00:38:46.107457+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.135502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"265'598","reported_seq":8477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813747+0000","last_change":"2024-04-24T00:38:37.645213+0000","last_active":"2024-04-24T00:38:46.813747+0000","last_peered":"2024-04-24T00:38:46.813747+0000","last_clean":"2024-04-24T00:38:46.813747+0000","last_became_active":"2024-04-24T00:38:37.638797+0000","last_became_peered":"2024-04-24T00:38:37.638797+0000","last_unstale":"2024-04-24T00:38:46.813747+0000","last_undegraded":"2024-04-24T00:38:46.813747+0000","last_fullsized":"2024-04-24T00:38:46.813747+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:20.508035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"265'1515","reported_seq":6441,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062859+0000","last_change":"2024-04-24T00:38:38.645480+0000","last_active":"2024-04-24T00:38:46.062859+0000","last_peered":"2024-04-24T00:38:46.062859+0000","last_clean":"2024-04-24T00:38:46.062859+0000","last_became_active":"2024-04-24T00:38:38.642317+0000","last_became_peered":"2024-04-24T00:38:38.642317+0000","last_unstale":"2024-04-24T00:38:46.062859+0000","last_undegraded":"2024-04-24T00:38:46.062859+0000","last_fullsized":"2024-04-24T00:38:46.062859+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:08.440907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"263'651","reported_seq":5104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813680+0000","last_change":"2024-04-24T00:38:32.655883+0000","last_active":"2024-04-24T00:38:46.813680+0000","last_peered":"2024-04-24T00:38:46.813680+0000","last_clean":"2024-04-24T00:38:46.813680+0000","last_became_active":"2024-04-24T00:38:32.650425+0000","last_became_peered":"2024-04-24T00:38:32.650425+0000","last_unstale":"2024-04-24T00:38:46.813680+0000","last_undegraded":"2024-04-24T00:38:46.813680+0000","last_fullsized":"2024-04-24T00:38:46.813680+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.249883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013270599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185642,"num_read_kb":1545553,"num_write":31215,"num_write_kb":1018173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":358075070,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},"log_size":22825,"ondisk_log_size":22825,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":2951840,"data_compressed":34925,"data_compressed_allocated":184320,"data_compressed_original":2949120,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56860,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314980,"statfs":{"total":96636764160,"available":96578539520,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66520,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305320,"statfs":{"total":96636764160,"available":96568647680,"internally_reserved":0,"allocated":3485696,"data_stored":1735171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74872,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296968,"statfs":{"total":96636764160,"available":96560095232,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92924,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94278916,"statfs":{"total":96636764160,"available":96541609984,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74856,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296984,"statfs":{"total":96636764160,"available":96560111616,"internally_reserved":0,"allocated":3497984,"data_stored":1741042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574915,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60968,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310872,"statfs":{"total":96636764160,"available":96574332928,"internally_reserved":0,"allocated":3497984,"data_stored":1741042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60980,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310860,"statfs":{"total":96636764160,"available":96574320640,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893164,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39516,"kb_used_data":3468,"kb_used_omap":6,"kb_used_meta":36025,"kb_avail":94332324,"statfs":{"total":96636764160,"available":96596299776,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":36890339},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:38:49.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:38:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:49.697 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:38:49.713 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-04-24T00:38:47.729265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":185758,"num_read_kb":1545653,"num_write":31374,"num_write_kb":1020589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8595,"num_bytes_recovered":360436542,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22942,"ondisk_log_size":22942,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":527496,"kb_used_data":27464,"kb_used_omap":17,"kb_used_meta":499886,"kb_avail":754447224,"statfs":{"total":773094113280,"available":772553957376,"internally_reserved":0,"allocated":28123136,"data_stored":15132594,"data_compressed":13970,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":17586,"internal_metadata":511884110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.857035"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":5999,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062969+0000","last_change":"2024-04-24T00:38:18.803340+0000","last_active":"2024-04-24T00:38:46.062969+0000","last_peered":"2024-04-24T00:38:46.062969+0000","last_clean":"2024-04-24T00:38:46.062969+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:46.062969+0000","last_undegraded":"2024-04-24T00:38:46.062969+0000","last_fullsized":"2024-04-24T00:38:46.062969+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:16.140487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"264'1050","reported_seq":5683,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370225+0000","last_change":"2024-04-24T00:38:36.641262+0000","last_active":"2024-04-24T00:38:46.370225+0000","last_peered":"2024-04-24T00:38:46.370225+0000","last_clean":"2024-04-24T00:38:46.370225+0000","last_became_active":"2024-04-24T00:38:36.639573+0000","last_became_peered":"2024-04-24T00:38:36.639573+0000","last_unstale":"2024-04-24T00:38:46.370225+0000","last_undegraded":"2024-04-24T00:38:46.370225+0000","last_fullsized":"2024-04-24T00:38:46.370225+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:18:26.284735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080331599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"263'591","reported_seq":7547,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911549+0000","last_change":"2024-04-24T00:38:33.652375+0000","last_active":"2024-04-24T00:38:45.911549+0000","last_peered":"2024-04-24T00:38:45.911549+0000","last_clean":"2024-04-24T00:38:45.911549+0000","last_became_active":"2024-04-24T00:38:33.649639+0000","last_became_peered":"2024-04-24T00:38:33.649639+0000","last_unstale":"2024-04-24T00:38:45.911549+0000","last_undegraded":"2024-04-24T00:38:45.911549+0000","last_fullsized":"2024-04-24T00:38:45.911549+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:36.602084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"266'795","reported_seq":6337,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:38:35.634698+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:38:35.628937+0000","last_became_peered":"2024-04-24T00:38:35.628937+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:48.395425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5266,"num_read_kb":61368,"num_write":1008,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"264'498","reported_seq":4556,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911438+0000","last_change":"2024-04-24T00:38:18.764647+0000","last_active":"2024-04-24T00:38:45.911438+0000","last_peered":"2024-04-24T00:38:45.911438+0000","last_clean":"2024-04-24T00:38:45.911438+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:45.911438+0000","last_undegraded":"2024-04-24T00:38:45.911438+0000","last_fullsized":"2024-04-24T00:38:45.911438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:59.132716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"265'390","reported_seq":6310,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910619+0000","last_change":"2024-04-24T00:38:35.638402+0000","last_active":"2024-04-24T00:38:45.910619+0000","last_peered":"2024-04-24T00:38:45.910619+0000","last_clean":"2024-04-24T00:38:45.910619+0000","last_became_active":"2024-04-24T00:38:35.631522+0000","last_became_peered":"2024-04-24T00:38:35.631522+0000","last_unstale":"2024-04-24T00:38:45.910619+0000","last_undegraded":"2024-04-24T00:38:45.910619+0000","last_fullsized":"2024-04-24T00:38:45.910619+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:44.645527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001287217,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4509,"num_read_kb":37330,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":3953416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"265'712","reported_seq":5909,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909836+0000","last_change":"2024-04-24T00:38:18.779600+0000","last_active":"2024-04-24T00:38:45.909836+0000","last_peered":"2024-04-24T00:38:45.909836+0000","last_clean":"2024-04-24T00:38:45.909836+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:45.909836+0000","last_undegraded":"2024-04-24T00:38:45.909836+0000","last_fullsized":"2024-04-24T00:38:45.909836+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:58.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059221100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"263'551","reported_seq":4877,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.369675+0000","last_change":"2024-04-24T00:38:36.632405+0000","last_active":"2024-04-24T00:38:46.369675+0000","last_peered":"2024-04-24T00:38:46.369675+0000","last_clean":"2024-04-24T00:38:46.369675+0000","last_became_active":"2024-04-24T00:38:36.631395+0000","last_became_peered":"2024-04-24T00:38:36.631395+0000","last_unstale":"2024-04-24T00:38:46.369675+0000","last_undegraded":"2024-04-24T00:38:46.369675+0000","last_fullsized":"2024-04-24T00:38:46.369675+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:35:30.536918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001058769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"263'339","reported_seq":4813,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063021+0000","last_change":"2024-04-24T00:38:18.803619+0000","last_active":"2024-04-24T00:38:46.063021+0000","last_peered":"2024-04-24T00:38:46.063021+0000","last_clean":"2024-04-24T00:38:46.063021+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:46.063021+0000","last_undegraded":"2024-04-24T00:38:46.063021+0000","last_fullsized":"2024-04-24T00:38:46.063021+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:53.921153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"266'751","reported_seq":6814,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370092+0000","last_change":"2024-04-24T00:38:18.756271+0000","last_active":"2024-04-24T00:38:46.370092+0000","last_peered":"2024-04-24T00:38:46.370092+0000","last_clean":"2024-04-24T00:38:46.370092+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:46.370092+0000","last_undegraded":"2024-04-24T00:38:46.370092+0000","last_fullsized":"2024-04-24T00:38:46.370092+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:36.855739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7650,"num_read_kb":50977,"num_write":1087,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"264'632","reported_seq":7275,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813710+0000","last_change":"2024-04-24T00:38:32.649992+0000","last_active":"2024-04-24T00:38:46.813710+0000","last_peered":"2024-04-24T00:38:46.813710+0000","last_clean":"2024-04-24T00:38:46.813710+0000","last_became_active":"2024-04-24T00:38:32.643949+0000","last_became_peered":"2024-04-24T00:38:32.643949+0000","last_unstale":"2024-04-24T00:38:46.813710+0000","last_undegraded":"2024-04-24T00:38:46.813710+0000","last_fullsized":"2024-04-24T00:38:46.813710+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:29:36.206394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"264'496","reported_seq":5383,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911030+0000","last_change":"2024-04-24T00:38:34.636361+0000","last_active":"2024-04-24T00:38:45.911030+0000","last_peered":"2024-04-24T00:38:45.911030+0000","last_clean":"2024-04-24T00:38:45.911030+0000","last_became_active":"2024-04-24T00:38:34.632609+0000","last_became_peered":"2024-04-24T00:38:34.632609+0000","last_unstale":"2024-04-24T00:38:45.911030+0000","last_undegraded":"2024-04-24T00:38:45.911030+0000","last_fullsized":"2024-04-24T00:38:45.911030+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:51:37.284131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010187200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":560,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107558+0000","last_change":"2024-04-24T00:38:30.696852+0000","last_active":"2024-04-24T00:38:46.107558+0000","last_peered":"2024-04-24T00:38:46.107558+0000","last_clean":"2024-04-24T00:38:46.107558+0000","last_became_active":"2024-04-24T00:38:30.696223+0000","last_became_peered":"2024-04-24T00:38:30.696223+0000","last_unstale":"2024-04-24T00:38:46.107558+0000","last_undegraded":"2024-04-24T00:38:46.107558+0000","last_fullsized":"2024-04-24T00:38:46.107558+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:08:35.422980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6523,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063089+0000","last_change":"2024-04-24T00:38:18.803861+0000","last_active":"2024-04-24T00:38:46.063089+0000","last_peered":"2024-04-24T00:38:46.063089+0000","last_clean":"2024-04-24T00:38:46.063089+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:46.063089+0000","last_undegraded":"2024-04-24T00:38:46.063089+0000","last_fullsized":"2024-04-24T00:38:46.063089+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:52.120677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910495+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:38:45.910495+0000","last_peered":"2024-04-24T00:38:45.910495+0000","last_clean":"2024-04-24T00:38:45.910495+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:45.910495+0000","last_undegraded":"2024-04-24T00:38:45.910495+0000","last_fullsized":"2024-04-24T00:38:45.910495+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4786,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370362+0000","last_change":"2024-04-24T00:38:36.639909+0000","last_active":"2024-04-24T00:38:46.370362+0000","last_peered":"2024-04-24T00:38:46.370362+0000","last_clean":"2024-04-24T00:38:46.370362+0000","last_became_active":"2024-04-24T00:38:36.637958+0000","last_became_peered":"2024-04-24T00:38:36.637958+0000","last_unstale":"2024-04-24T00:38:46.370362+0000","last_undegraded":"2024-04-24T00:38:46.370362+0000","last_fullsized":"2024-04-24T00:38:46.370362+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:23.662984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017453659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813411+0000","last_change":"2024-04-24T00:38:30.641161+0000","last_active":"2024-04-24T00:38:46.813411+0000","last_peered":"2024-04-24T00:38:46.813411+0000","last_clean":"2024-04-24T00:38:46.813411+0000","last_became_active":"2024-04-24T00:38:30.640288+0000","last_became_peered":"2024-04-24T00:38:30.640288+0000","last_unstale":"2024-04-24T00:38:46.813411+0000","last_undegraded":"2024-04-24T00:38:46.813411+0000","last_fullsized":"2024-04-24T00:38:46.813411+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:46:24.491722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4161,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910063+0000","last_change":"2024-04-24T00:38:18.780633+0000","last_active":"2024-04-24T00:38:45.910063+0000","last_peered":"2024-04-24T00:38:45.910063+0000","last_clean":"2024-04-24T00:38:45.910063+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:45.910063+0000","last_undegraded":"2024-04-24T00:38:45.910063+0000","last_fullsized":"2024-04-24T00:38:45.910063+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:46.783523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099788599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909769+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:38:45.909769+0000","last_peered":"2024-04-24T00:38:45.909769+0000","last_clean":"2024-04-24T00:38:45.909769+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:45.909769+0000","last_undegraded":"2024-04-24T00:38:45.909769+0000","last_fullsized":"2024-04-24T00:38:45.909769+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6292,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911342+0000","last_change":"2024-04-24T00:38:18.767450+0000","last_active":"2024-04-24T00:38:45.911342+0000","last_peered":"2024-04-24T00:38:45.911342+0000","last_clean":"2024-04-24T00:38:45.911342+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:45.911342+0000","last_undegraded":"2024-04-24T00:38:45.911342+0000","last_fullsized":"2024-04-24T00:38:45.911342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:09:34.691573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"266'580","reported_seq":6804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910527+0000","last_change":"2024-04-24T00:38:32.649496+0000","last_active":"2024-04-24T00:38:45.910527+0000","last_peered":"2024-04-24T00:38:45.910527+0000","last_clean":"2024-04-24T00:38:45.910527+0000","last_became_active":"2024-04-24T00:38:32.645508+0000","last_became_peered":"2024-04-24T00:38:32.645508+0000","last_unstale":"2024-04-24T00:38:45.910527+0000","last_undegraded":"2024-04-24T00:38:45.910527+0000","last_fullsized":"2024-04-24T00:38:45.910527+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:59.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910983+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:38:45.910983+0000","last_peered":"2024-04-24T00:38:45.910983+0000","last_clean":"2024-04-24T00:38:45.910983+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:45.910983+0000","last_undegraded":"2024-04-24T00:38:45.910983+0000","last_fullsized":"2024-04-24T00:38:45.910983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7975,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909664+0000","last_change":"2024-04-24T00:38:18.780209+0000","last_active":"2024-04-24T00:38:45.909664+0000","last_peered":"2024-04-24T00:38:45.909664+0000","last_clean":"2024-04-24T00:38:45.909664+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:45.909664+0000","last_undegraded":"2024-04-24T00:38:45.909664+0000","last_fullsized":"2024-04-24T00:38:45.909664+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:49.181426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":550,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370389+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:38:46.370389+0000","last_peered":"2024-04-24T00:38:46.370389+0000","last_clean":"2024-04-24T00:38:46.370389+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:46.370389+0000","last_undegraded":"2024-04-24T00:38:46.370389+0000","last_fullsized":"2024-04-24T00:38:46.370389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":5035,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063003+0000","last_change":"2024-04-24T00:38:34.632199+0000","last_active":"2024-04-24T00:38:46.063003+0000","last_peered":"2024-04-24T00:38:46.063003+0000","last_clean":"2024-04-24T00:38:46.063003+0000","last_became_active":"2024-04-24T00:38:34.629456+0000","last_became_peered":"2024-04-24T00:38:34.629456+0000","last_unstale":"2024-04-24T00:38:46.063003+0000","last_undegraded":"2024-04-24T00:38:46.063003+0000","last_fullsized":"2024-04-24T00:38:46.063003+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:05:27.920476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813416+0000","last_change":"2024-04-24T00:38:30.634583+0000","last_active":"2024-04-24T00:38:46.813416+0000","last_peered":"2024-04-24T00:38:46.813416+0000","last_clean":"2024-04-24T00:38:46.813416+0000","last_became_active":"2024-04-24T00:38:30.633606+0000","last_became_peered":"2024-04-24T00:38:30.633606+0000","last_unstale":"2024-04-24T00:38:46.813416+0000","last_undegraded":"2024-04-24T00:38:46.813416+0000","last_fullsized":"2024-04-24T00:38:46.813416+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:24:57.510774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2873,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.369748+0000","last_change":"2024-04-24T00:38:36.633952+0000","last_active":"2024-04-24T00:38:46.369748+0000","last_peered":"2024-04-24T00:38:46.369748+0000","last_clean":"2024-04-24T00:38:46.369748+0000","last_became_active":"2024-04-24T00:38:36.632239+0000","last_became_peered":"2024-04-24T00:38:36.632239+0000","last_unstale":"2024-04-24T00:38:46.369748+0000","last_undegraded":"2024-04-24T00:38:46.369748+0000","last_fullsized":"2024-04-24T00:38:46.369748+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:50.888554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012896030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":506,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813566+0000","last_change":"2024-04-24T00:38:36.642364+0000","last_active":"2024-04-24T00:38:46.813566+0000","last_peered":"2024-04-24T00:38:46.813566+0000","last_clean":"2024-04-24T00:38:46.813566+0000","last_became_active":"2024-04-24T00:38:36.641100+0000","last_became_peered":"2024-04-24T00:38:36.641100+0000","last_unstale":"2024-04-24T00:38:46.813566+0000","last_undegraded":"2024-04-24T00:38:46.813566+0000","last_fullsized":"2024-04-24T00:38:46.813566+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:00.586831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6983,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.369772+0000","last_change":"2024-04-24T00:38:18.755623+0000","last_active":"2024-04-24T00:38:46.369772+0000","last_peered":"2024-04-24T00:38:46.369772+0000","last_clean":"2024-04-24T00:38:46.369772+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:46.369772+0000","last_undegraded":"2024-04-24T00:38:46.369772+0000","last_fullsized":"2024-04-24T00:38:46.369772+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:44.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"265'987","reported_seq":5319,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910553+0000","last_change":"2024-04-24T00:38:33.656528+0000","last_active":"2024-04-24T00:38:45.910553+0000","last_peered":"2024-04-24T00:38:45.910553+0000","last_clean":"2024-04-24T00:38:45.910553+0000","last_became_active":"2024-04-24T00:38:33.650599+0000","last_became_peered":"2024-04-24T00:38:33.650599+0000","last_unstale":"2024-04-24T00:38:45.910553+0000","last_undegraded":"2024-04-24T00:38:45.910553+0000","last_fullsized":"2024-04-24T00:38:45.910553+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:19.834939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"263'1904","reported_seq":4804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107528+0000","last_change":"2024-04-24T00:38:34.638496+0000","last_active":"2024-04-24T00:38:46.107528+0000","last_peered":"2024-04-24T00:38:46.107528+0000","last_clean":"2024-04-24T00:38:46.107528+0000","last_became_active":"2024-04-24T00:38:34.634199+0000","last_became_peered":"2024-04-24T00:38:34.634199+0000","last_unstale":"2024-04-24T00:38:46.107528+0000","last_undegraded":"2024-04-24T00:38:46.107528+0000","last_fullsized":"2024-04-24T00:38:46.107528+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:57.455108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001715414,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"263'461","reported_seq":4301,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.369928+0000","last_change":"2024-04-24T00:38:36.634741+0000","last_active":"2024-04-24T00:38:46.369928+0000","last_peered":"2024-04-24T00:38:46.369928+0000","last_clean":"2024-04-24T00:38:46.369928+0000","last_became_active":"2024-04-24T00:38:36.633468+0000","last_became_peered":"2024-04-24T00:38:36.633468+0000","last_unstale":"2024-04-24T00:38:46.369928+0000","last_undegraded":"2024-04-24T00:38:46.369928+0000","last_fullsized":"2024-04-24T00:38:46.369928+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:40.363226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002035177,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"266'831","reported_seq":10117,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911887+0000","last_change":"2024-04-24T00:38:18.766586+0000","last_active":"2024-04-24T00:38:45.911887+0000","last_peered":"2024-04-24T00:38:45.911887+0000","last_clean":"2024-04-24T00:38:45.911887+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:45.911887+0000","last_undegraded":"2024-04-24T00:38:45.911887+0000","last_fullsized":"2024-04-24T00:38:45.911887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:15.463491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10600,"num_read_kb":68795,"num_write":1180,"num_write_kb":35219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"263'426","reported_seq":2956,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813960+0000","last_change":"2024-04-24T00:38:35.637609+0000","last_active":"2024-04-24T00:38:46.813960+0000","last_peered":"2024-04-24T00:38:46.813960+0000","last_clean":"2024-04-24T00:38:46.813960+0000","last_became_active":"2024-04-24T00:38:35.632277+0000","last_became_peered":"2024-04-24T00:38:35.632277+0000","last_unstale":"2024-04-24T00:38:46.813960+0000","last_undegraded":"2024-04-24T00:38:46.813960+0000","last_fullsized":"2024-04-24T00:38:46.813960+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:03.305609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"266'671","reported_seq":8093,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814115+0000","last_change":"2024-04-24T00:38:37.642821+0000","last_active":"2024-04-24T00:38:46.814115+0000","last_peered":"2024-04-24T00:38:46.814115+0000","last_clean":"2024-04-24T00:38:46.814115+0000","last_became_active":"2024-04-24T00:38:37.638998+0000","last_became_peered":"2024-04-24T00:38:37.638998+0000","last_unstale":"2024-04-24T00:38:46.814115+0000","last_undegraded":"2024-04-24T00:38:46.814115+0000","last_fullsized":"2024-04-24T00:38:46.814115+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:10:52.647284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001218907,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":952,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"263'874","reported_seq":5902,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814247+0000","last_change":"2024-04-24T00:38:36.646741+0000","last_active":"2024-04-24T00:38:46.814247+0000","last_peered":"2024-04-24T00:38:46.814247+0000","last_clean":"2024-04-24T00:38:46.814247+0000","last_became_active":"2024-04-24T00:38:36.640462+0000","last_became_peered":"2024-04-24T00:38:36.640462+0000","last_unstale":"2024-04-24T00:38:46.814247+0000","last_undegraded":"2024-04-24T00:38:46.814247+0000","last_fullsized":"2024-04-24T00:38:46.814247+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:25.891296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"265'610","reported_seq":6427,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107457+0000","last_change":"2024-04-24T00:38:18.844983+0000","last_active":"2024-04-24T00:38:46.107457+0000","last_peered":"2024-04-24T00:38:46.107457+0000","last_clean":"2024-04-24T00:38:46.107457+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:46.107457+0000","last_undegraded":"2024-04-24T00:38:46.107457+0000","last_fullsized":"2024-04-24T00:38:46.107457+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.135502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"265'598","reported_seq":8477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813747+0000","last_change":"2024-04-24T00:38:37.645213+0000","last_active":"2024-04-24T00:38:46.813747+0000","last_peered":"2024-04-24T00:38:46.813747+0000","last_clean":"2024-04-24T00:38:46.813747+0000","last_became_active":"2024-04-24T00:38:37.638797+0000","last_became_peered":"2024-04-24T00:38:37.638797+0000","last_unstale":"2024-04-24T00:38:46.813747+0000","last_undegraded":"2024-04-24T00:38:46.813747+0000","last_fullsized":"2024-04-24T00:38:46.813747+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:20.508035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"265'1515","reported_seq":6441,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062859+0000","last_change":"2024-04-24T00:38:38.645480+0000","last_active":"2024-04-24T00:38:46.062859+0000","last_peered":"2024-04-24T00:38:46.062859+0000","last_clean":"2024-04-24T00:38:46.062859+0000","last_became_active":"2024-04-24T00:38:38.642317+0000","last_became_peered":"2024-04-24T00:38:38.642317+0000","last_unstale":"2024-04-24T00:38:46.062859+0000","last_undegraded":"2024-04-24T00:38:46.062859+0000","last_fullsized":"2024-04-24T00:38:46.062859+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:08.440907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"263'651","reported_seq":5104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813680+0000","last_change":"2024-04-24T00:38:32.655883+0000","last_active":"2024-04-24T00:38:46.813680+0000","last_peered":"2024-04-24T00:38:46.813680+0000","last_clean":"2024-04-24T00:38:46.813680+0000","last_became_active":"2024-04-24T00:38:32.650425+0000","last_became_peered":"2024-04-24T00:38:32.650425+0000","last_unstale":"2024-04-24T00:38:46.813680+0000","last_undegraded":"2024-04-24T00:38:46.813680+0000","last_fullsized":"2024-04-24T00:38:46.813680+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.249883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013270599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185642,"num_read_kb":1545553,"num_write":31215,"num_write_kb":1018173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":358075070,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},"log_size":22825,"ondisk_log_size":22825,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":2951840,"data_compressed":34925,"data_compressed_allocated":184320,"data_compressed_original":2949120,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56860,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314980,"statfs":{"total":96636764160,"available":96578539520,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66520,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305320,"statfs":{"total":96636764160,"available":96568647680,"internally_reserved":0,"allocated":3485696,"data_stored":1735171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74872,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296968,"statfs":{"total":96636764160,"available":96560095232,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92924,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94278916,"statfs":{"total":96636764160,"available":96541609984,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74856,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296984,"statfs":{"total":96636764160,"available":96560111616,"internally_reserved":0,"allocated":3497984,"data_stored":1741042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574915,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60968,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310872,"statfs":{"total":96636764160,"available":96574332928,"internally_reserved":0,"allocated":3497984,"data_stored":1741042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60980,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310860,"statfs":{"total":96636764160,"available":96574320640,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893164,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39516,"kb_used_data":3468,"kb_used_omap":6,"kb_used_meta":36025,"kb_avail":94332324,"statfs":{"total":96636764160,"available":96596299776,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":36890339},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:38:49.715 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T00:38:49.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-24T00:38:50.035 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T00:38:50.047 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T00:38:50.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:38:50.334 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:50.334 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:38:50.350 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-04-24T00:38:49.730029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":185758,"num_read_kb":1545653,"num_write":31374,"num_write_kb":1020589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8595,"num_bytes_recovered":360436542,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22942,"ondisk_log_size":22942,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":527540,"kb_used_data":27512,"kb_used_omap":17,"kb_used_meta":499886,"kb_avail":754447180,"statfs":{"total":773094113280,"available":772553912320,"internally_reserved":0,"allocated":28172288,"data_stored":15156078,"data_compressed":13970,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":17586,"internal_metadata":511884110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002013"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":5999,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062969+0000","last_change":"2024-04-24T00:38:18.803340+0000","last_active":"2024-04-24T00:38:46.062969+0000","last_peered":"2024-04-24T00:38:46.062969+0000","last_clean":"2024-04-24T00:38:46.062969+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:46.062969+0000","last_undegraded":"2024-04-24T00:38:46.062969+0000","last_fullsized":"2024-04-24T00:38:46.062969+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:16.140487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"264'1050","reported_seq":5685,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440757+0000","last_change":"2024-04-24T00:38:36.641262+0000","last_active":"2024-04-24T00:38:47.440757+0000","last_peered":"2024-04-24T00:38:47.440757+0000","last_clean":"2024-04-24T00:38:47.440757+0000","last_became_active":"2024-04-24T00:38:36.639573+0000","last_became_peered":"2024-04-24T00:38:36.639573+0000","last_unstale":"2024-04-24T00:38:47.440757+0000","last_undegraded":"2024-04-24T00:38:47.440757+0000","last_fullsized":"2024-04-24T00:38:47.440757+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:18:26.284735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080331599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"263'591","reported_seq":7547,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911549+0000","last_change":"2024-04-24T00:38:33.652375+0000","last_active":"2024-04-24T00:38:45.911549+0000","last_peered":"2024-04-24T00:38:45.911549+0000","last_clean":"2024-04-24T00:38:45.911549+0000","last_became_active":"2024-04-24T00:38:33.649639+0000","last_became_peered":"2024-04-24T00:38:33.649639+0000","last_unstale":"2024-04-24T00:38:45.911549+0000","last_undegraded":"2024-04-24T00:38:45.911549+0000","last_fullsized":"2024-04-24T00:38:45.911549+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:36.602084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"266'795","reported_seq":6337,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:38:35.634698+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:38:35.628937+0000","last_became_peered":"2024-04-24T00:38:35.628937+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:48.395425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5266,"num_read_kb":61368,"num_write":1008,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"264'498","reported_seq":4556,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911438+0000","last_change":"2024-04-24T00:38:18.764647+0000","last_active":"2024-04-24T00:38:45.911438+0000","last_peered":"2024-04-24T00:38:45.911438+0000","last_clean":"2024-04-24T00:38:45.911438+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:45.911438+0000","last_undegraded":"2024-04-24T00:38:45.911438+0000","last_fullsized":"2024-04-24T00:38:45.911438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:59.132716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"265'390","reported_seq":6310,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910619+0000","last_change":"2024-04-24T00:38:35.638402+0000","last_active":"2024-04-24T00:38:45.910619+0000","last_peered":"2024-04-24T00:38:45.910619+0000","last_clean":"2024-04-24T00:38:45.910619+0000","last_became_active":"2024-04-24T00:38:35.631522+0000","last_became_peered":"2024-04-24T00:38:35.631522+0000","last_unstale":"2024-04-24T00:38:45.910619+0000","last_undegraded":"2024-04-24T00:38:45.910619+0000","last_fullsized":"2024-04-24T00:38:45.910619+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:44.645527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001287217,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4509,"num_read_kb":37330,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":3953416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"265'712","reported_seq":5909,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909836+0000","last_change":"2024-04-24T00:38:18.779600+0000","last_active":"2024-04-24T00:38:45.909836+0000","last_peered":"2024-04-24T00:38:45.909836+0000","last_clean":"2024-04-24T00:38:45.909836+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:45.909836+0000","last_undegraded":"2024-04-24T00:38:45.909836+0000","last_fullsized":"2024-04-24T00:38:45.909836+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:58.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059221100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"263'551","reported_seq":4879,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440456+0000","last_change":"2024-04-24T00:38:36.632405+0000","last_active":"2024-04-24T00:38:47.440456+0000","last_peered":"2024-04-24T00:38:47.440456+0000","last_clean":"2024-04-24T00:38:47.440456+0000","last_became_active":"2024-04-24T00:38:36.631395+0000","last_became_peered":"2024-04-24T00:38:36.631395+0000","last_unstale":"2024-04-24T00:38:47.440456+0000","last_undegraded":"2024-04-24T00:38:47.440456+0000","last_fullsized":"2024-04-24T00:38:47.440456+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:35:30.536918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001058769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"263'339","reported_seq":4813,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063021+0000","last_change":"2024-04-24T00:38:18.803619+0000","last_active":"2024-04-24T00:38:46.063021+0000","last_peered":"2024-04-24T00:38:46.063021+0000","last_clean":"2024-04-24T00:38:46.063021+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:46.063021+0000","last_undegraded":"2024-04-24T00:38:46.063021+0000","last_fullsized":"2024-04-24T00:38:46.063021+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:53.921153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"266'751","reported_seq":6816,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440709+0000","last_change":"2024-04-24T00:38:18.756271+0000","last_active":"2024-04-24T00:38:47.440709+0000","last_peered":"2024-04-24T00:38:47.440709+0000","last_clean":"2024-04-24T00:38:47.440709+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:47.440709+0000","last_undegraded":"2024-04-24T00:38:47.440709+0000","last_fullsized":"2024-04-24T00:38:47.440709+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:36.855739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7650,"num_read_kb":50977,"num_write":1087,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"264'632","reported_seq":7275,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813710+0000","last_change":"2024-04-24T00:38:32.649992+0000","last_active":"2024-04-24T00:38:46.813710+0000","last_peered":"2024-04-24T00:38:46.813710+0000","last_clean":"2024-04-24T00:38:46.813710+0000","last_became_active":"2024-04-24T00:38:32.643949+0000","last_became_peered":"2024-04-24T00:38:32.643949+0000","last_unstale":"2024-04-24T00:38:46.813710+0000","last_undegraded":"2024-04-24T00:38:46.813710+0000","last_fullsized":"2024-04-24T00:38:46.813710+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:29:36.206394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"264'496","reported_seq":5383,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911030+0000","last_change":"2024-04-24T00:38:34.636361+0000","last_active":"2024-04-24T00:38:45.911030+0000","last_peered":"2024-04-24T00:38:45.911030+0000","last_clean":"2024-04-24T00:38:45.911030+0000","last_became_active":"2024-04-24T00:38:34.632609+0000","last_became_peered":"2024-04-24T00:38:34.632609+0000","last_unstale":"2024-04-24T00:38:45.911030+0000","last_undegraded":"2024-04-24T00:38:45.911030+0000","last_fullsized":"2024-04-24T00:38:45.911030+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:51:37.284131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010187200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":560,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107558+0000","last_change":"2024-04-24T00:38:30.696852+0000","last_active":"2024-04-24T00:38:46.107558+0000","last_peered":"2024-04-24T00:38:46.107558+0000","last_clean":"2024-04-24T00:38:46.107558+0000","last_became_active":"2024-04-24T00:38:30.696223+0000","last_became_peered":"2024-04-24T00:38:30.696223+0000","last_unstale":"2024-04-24T00:38:46.107558+0000","last_undegraded":"2024-04-24T00:38:46.107558+0000","last_fullsized":"2024-04-24T00:38:46.107558+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:08:35.422980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6523,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063089+0000","last_change":"2024-04-24T00:38:18.803861+0000","last_active":"2024-04-24T00:38:46.063089+0000","last_peered":"2024-04-24T00:38:46.063089+0000","last_clean":"2024-04-24T00:38:46.063089+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:46.063089+0000","last_undegraded":"2024-04-24T00:38:46.063089+0000","last_fullsized":"2024-04-24T00:38:46.063089+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:52.120677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910495+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:38:45.910495+0000","last_peered":"2024-04-24T00:38:45.910495+0000","last_clean":"2024-04-24T00:38:45.910495+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:45.910495+0000","last_undegraded":"2024-04-24T00:38:45.910495+0000","last_fullsized":"2024-04-24T00:38:45.910495+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4788,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440816+0000","last_change":"2024-04-24T00:38:36.639909+0000","last_active":"2024-04-24T00:38:47.440816+0000","last_peered":"2024-04-24T00:38:47.440816+0000","last_clean":"2024-04-24T00:38:47.440816+0000","last_became_active":"2024-04-24T00:38:36.637958+0000","last_became_peered":"2024-04-24T00:38:36.637958+0000","last_unstale":"2024-04-24T00:38:47.440816+0000","last_undegraded":"2024-04-24T00:38:47.440816+0000","last_fullsized":"2024-04-24T00:38:47.440816+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:23.662984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017453659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813411+0000","last_change":"2024-04-24T00:38:30.641161+0000","last_active":"2024-04-24T00:38:46.813411+0000","last_peered":"2024-04-24T00:38:46.813411+0000","last_clean":"2024-04-24T00:38:46.813411+0000","last_became_active":"2024-04-24T00:38:30.640288+0000","last_became_peered":"2024-04-24T00:38:30.640288+0000","last_unstale":"2024-04-24T00:38:46.813411+0000","last_undegraded":"2024-04-24T00:38:46.813411+0000","last_fullsized":"2024-04-24T00:38:46.813411+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:46:24.491722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4161,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910063+0000","last_change":"2024-04-24T00:38:18.780633+0000","last_active":"2024-04-24T00:38:45.910063+0000","last_peered":"2024-04-24T00:38:45.910063+0000","last_clean":"2024-04-24T00:38:45.910063+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:45.910063+0000","last_undegraded":"2024-04-24T00:38:45.910063+0000","last_fullsized":"2024-04-24T00:38:45.910063+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:46.783523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099788599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909769+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:38:45.909769+0000","last_peered":"2024-04-24T00:38:45.909769+0000","last_clean":"2024-04-24T00:38:45.909769+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:45.909769+0000","last_undegraded":"2024-04-24T00:38:45.909769+0000","last_fullsized":"2024-04-24T00:38:45.909769+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6292,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911342+0000","last_change":"2024-04-24T00:38:18.767450+0000","last_active":"2024-04-24T00:38:45.911342+0000","last_peered":"2024-04-24T00:38:45.911342+0000","last_clean":"2024-04-24T00:38:45.911342+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:45.911342+0000","last_undegraded":"2024-04-24T00:38:45.911342+0000","last_fullsized":"2024-04-24T00:38:45.911342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:09:34.691573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"266'580","reported_seq":6804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910527+0000","last_change":"2024-04-24T00:38:32.649496+0000","last_active":"2024-04-24T00:38:45.910527+0000","last_peered":"2024-04-24T00:38:45.910527+0000","last_clean":"2024-04-24T00:38:45.910527+0000","last_became_active":"2024-04-24T00:38:32.645508+0000","last_became_peered":"2024-04-24T00:38:32.645508+0000","last_unstale":"2024-04-24T00:38:45.910527+0000","last_undegraded":"2024-04-24T00:38:45.910527+0000","last_fullsized":"2024-04-24T00:38:45.910527+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:59.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910983+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:38:45.910983+0000","last_peered":"2024-04-24T00:38:45.910983+0000","last_clean":"2024-04-24T00:38:45.910983+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:45.910983+0000","last_undegraded":"2024-04-24T00:38:45.910983+0000","last_fullsized":"2024-04-24T00:38:45.910983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7975,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909664+0000","last_change":"2024-04-24T00:38:18.780209+0000","last_active":"2024-04-24T00:38:45.909664+0000","last_peered":"2024-04-24T00:38:45.909664+0000","last_clean":"2024-04-24T00:38:45.909664+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:45.909664+0000","last_undegraded":"2024-04-24T00:38:45.909664+0000","last_fullsized":"2024-04-24T00:38:45.909664+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:49.181426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":550,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370389+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:38:46.370389+0000","last_peered":"2024-04-24T00:38:46.370389+0000","last_clean":"2024-04-24T00:38:46.370389+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:46.370389+0000","last_undegraded":"2024-04-24T00:38:46.370389+0000","last_fullsized":"2024-04-24T00:38:46.370389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":5035,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063003+0000","last_change":"2024-04-24T00:38:34.632199+0000","last_active":"2024-04-24T00:38:46.063003+0000","last_peered":"2024-04-24T00:38:46.063003+0000","last_clean":"2024-04-24T00:38:46.063003+0000","last_became_active":"2024-04-24T00:38:34.629456+0000","last_became_peered":"2024-04-24T00:38:34.629456+0000","last_unstale":"2024-04-24T00:38:46.063003+0000","last_undegraded":"2024-04-24T00:38:46.063003+0000","last_fullsized":"2024-04-24T00:38:46.063003+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:05:27.920476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813416+0000","last_change":"2024-04-24T00:38:30.634583+0000","last_active":"2024-04-24T00:38:46.813416+0000","last_peered":"2024-04-24T00:38:46.813416+0000","last_clean":"2024-04-24T00:38:46.813416+0000","last_became_active":"2024-04-24T00:38:30.633606+0000","last_became_peered":"2024-04-24T00:38:30.633606+0000","last_unstale":"2024-04-24T00:38:46.813416+0000","last_undegraded":"2024-04-24T00:38:46.813416+0000","last_fullsized":"2024-04-24T00:38:46.813416+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:24:57.510774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2875,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440503+0000","last_change":"2024-04-24T00:38:36.633952+0000","last_active":"2024-04-24T00:38:47.440503+0000","last_peered":"2024-04-24T00:38:47.440503+0000","last_clean":"2024-04-24T00:38:47.440503+0000","last_became_active":"2024-04-24T00:38:36.632239+0000","last_became_peered":"2024-04-24T00:38:36.632239+0000","last_unstale":"2024-04-24T00:38:47.440503+0000","last_undegraded":"2024-04-24T00:38:47.440503+0000","last_fullsized":"2024-04-24T00:38:47.440503+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:50.888554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012896030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":506,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813566+0000","last_change":"2024-04-24T00:38:36.642364+0000","last_active":"2024-04-24T00:38:46.813566+0000","last_peered":"2024-04-24T00:38:46.813566+0000","last_clean":"2024-04-24T00:38:46.813566+0000","last_became_active":"2024-04-24T00:38:36.641100+0000","last_became_peered":"2024-04-24T00:38:36.641100+0000","last_unstale":"2024-04-24T00:38:46.813566+0000","last_undegraded":"2024-04-24T00:38:46.813566+0000","last_fullsized":"2024-04-24T00:38:46.813566+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:00.586831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6985,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440455+0000","last_change":"2024-04-24T00:38:18.755623+0000","last_active":"2024-04-24T00:38:47.440455+0000","last_peered":"2024-04-24T00:38:47.440455+0000","last_clean":"2024-04-24T00:38:47.440455+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:47.440455+0000","last_undegraded":"2024-04-24T00:38:47.440455+0000","last_fullsized":"2024-04-24T00:38:47.440455+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:44.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"265'987","reported_seq":5319,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910553+0000","last_change":"2024-04-24T00:38:33.656528+0000","last_active":"2024-04-24T00:38:45.910553+0000","last_peered":"2024-04-24T00:38:45.910553+0000","last_clean":"2024-04-24T00:38:45.910553+0000","last_became_active":"2024-04-24T00:38:33.650599+0000","last_became_peered":"2024-04-24T00:38:33.650599+0000","last_unstale":"2024-04-24T00:38:45.910553+0000","last_undegraded":"2024-04-24T00:38:45.910553+0000","last_fullsized":"2024-04-24T00:38:45.910553+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:19.834939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"263'1904","reported_seq":4804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107528+0000","last_change":"2024-04-24T00:38:34.638496+0000","last_active":"2024-04-24T00:38:46.107528+0000","last_peered":"2024-04-24T00:38:46.107528+0000","last_clean":"2024-04-24T00:38:46.107528+0000","last_became_active":"2024-04-24T00:38:34.634199+0000","last_became_peered":"2024-04-24T00:38:34.634199+0000","last_unstale":"2024-04-24T00:38:46.107528+0000","last_undegraded":"2024-04-24T00:38:46.107528+0000","last_fullsized":"2024-04-24T00:38:46.107528+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:57.455108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001715414,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"263'461","reported_seq":4303,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440544+0000","last_change":"2024-04-24T00:38:36.634741+0000","last_active":"2024-04-24T00:38:47.440544+0000","last_peered":"2024-04-24T00:38:47.440544+0000","last_clean":"2024-04-24T00:38:47.440544+0000","last_became_active":"2024-04-24T00:38:36.633468+0000","last_became_peered":"2024-04-24T00:38:36.633468+0000","last_unstale":"2024-04-24T00:38:47.440544+0000","last_undegraded":"2024-04-24T00:38:47.440544+0000","last_fullsized":"2024-04-24T00:38:47.440544+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:40.363226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002035177,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"266'831","reported_seq":10117,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911887+0000","last_change":"2024-04-24T00:38:18.766586+0000","last_active":"2024-04-24T00:38:45.911887+0000","last_peered":"2024-04-24T00:38:45.911887+0000","last_clean":"2024-04-24T00:38:45.911887+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:45.911887+0000","last_undegraded":"2024-04-24T00:38:45.911887+0000","last_fullsized":"2024-04-24T00:38:45.911887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:15.463491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10600,"num_read_kb":68795,"num_write":1180,"num_write_kb":35219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"263'426","reported_seq":2956,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813960+0000","last_change":"2024-04-24T00:38:35.637609+0000","last_active":"2024-04-24T00:38:46.813960+0000","last_peered":"2024-04-24T00:38:46.813960+0000","last_clean":"2024-04-24T00:38:46.813960+0000","last_became_active":"2024-04-24T00:38:35.632277+0000","last_became_peered":"2024-04-24T00:38:35.632277+0000","last_unstale":"2024-04-24T00:38:46.813960+0000","last_undegraded":"2024-04-24T00:38:46.813960+0000","last_fullsized":"2024-04-24T00:38:46.813960+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:03.305609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"266'671","reported_seq":8093,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814115+0000","last_change":"2024-04-24T00:38:37.642821+0000","last_active":"2024-04-24T00:38:46.814115+0000","last_peered":"2024-04-24T00:38:46.814115+0000","last_clean":"2024-04-24T00:38:46.814115+0000","last_became_active":"2024-04-24T00:38:37.638998+0000","last_became_peered":"2024-04-24T00:38:37.638998+0000","last_unstale":"2024-04-24T00:38:46.814115+0000","last_undegraded":"2024-04-24T00:38:46.814115+0000","last_fullsized":"2024-04-24T00:38:46.814115+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:10:52.647284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001218907,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":952,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"263'874","reported_seq":5902,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814247+0000","last_change":"2024-04-24T00:38:36.646741+0000","last_active":"2024-04-24T00:38:46.814247+0000","last_peered":"2024-04-24T00:38:46.814247+0000","last_clean":"2024-04-24T00:38:46.814247+0000","last_became_active":"2024-04-24T00:38:36.640462+0000","last_became_peered":"2024-04-24T00:38:36.640462+0000","last_unstale":"2024-04-24T00:38:46.814247+0000","last_undegraded":"2024-04-24T00:38:46.814247+0000","last_fullsized":"2024-04-24T00:38:46.814247+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:25.891296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"265'610","reported_seq":6427,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107457+0000","last_change":"2024-04-24T00:38:18.844983+0000","last_active":"2024-04-24T00:38:46.107457+0000","last_peered":"2024-04-24T00:38:46.107457+0000","last_clean":"2024-04-24T00:38:46.107457+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:46.107457+0000","last_undegraded":"2024-04-24T00:38:46.107457+0000","last_fullsized":"2024-04-24T00:38:46.107457+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.135502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"265'598","reported_seq":8477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813747+0000","last_change":"2024-04-24T00:38:37.645213+0000","last_active":"2024-04-24T00:38:46.813747+0000","last_peered":"2024-04-24T00:38:46.813747+0000","last_clean":"2024-04-24T00:38:46.813747+0000","last_became_active":"2024-04-24T00:38:37.638797+0000","last_became_peered":"2024-04-24T00:38:37.638797+0000","last_unstale":"2024-04-24T00:38:46.813747+0000","last_undegraded":"2024-04-24T00:38:46.813747+0000","last_fullsized":"2024-04-24T00:38:46.813747+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:20.508035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"265'1515","reported_seq":6441,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062859+0000","last_change":"2024-04-24T00:38:38.645480+0000","last_active":"2024-04-24T00:38:46.062859+0000","last_peered":"2024-04-24T00:38:46.062859+0000","last_clean":"2024-04-24T00:38:46.062859+0000","last_became_active":"2024-04-24T00:38:38.642317+0000","last_became_peered":"2024-04-24T00:38:38.642317+0000","last_unstale":"2024-04-24T00:38:46.062859+0000","last_undegraded":"2024-04-24T00:38:46.062859+0000","last_fullsized":"2024-04-24T00:38:46.062859+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:08.440907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"263'651","reported_seq":5104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813680+0000","last_change":"2024-04-24T00:38:32.655883+0000","last_active":"2024-04-24T00:38:46.813680+0000","last_peered":"2024-04-24T00:38:46.813680+0000","last_clean":"2024-04-24T00:38:46.813680+0000","last_became_active":"2024-04-24T00:38:32.650425+0000","last_became_peered":"2024-04-24T00:38:32.650425+0000","last_unstale":"2024-04-24T00:38:46.813680+0000","last_undegraded":"2024-04-24T00:38:46.813680+0000","last_fullsized":"2024-04-24T00:38:46.813680+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.249883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013270599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185642,"num_read_kb":1545553,"num_write":31215,"num_write_kb":1018173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":358075070,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},"log_size":22825,"ondisk_log_size":22825,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":2951840,"data_compressed":34925,"data_compressed_allocated":184320,"data_compressed_original":2949120,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56860,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314980,"statfs":{"total":96636764160,"available":96578539520,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66548,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305292,"statfs":{"total":96636764160,"available":96568619008,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74872,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296968,"statfs":{"total":96636764160,"available":96560095232,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92924,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94278916,"statfs":{"total":96636764160,"available":96541609984,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74864,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296976,"statfs":{"total":96636764160,"available":96560103424,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60976,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310864,"statfs":{"total":96636764160,"available":96574324736,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60980,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310860,"statfs":{"total":96636764160,"available":96574320640,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893165,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39516,"kb_used_data":3468,"kb_used_omap":6,"kb_used_meta":36025,"kb_avail":94332324,"statfs":{"total":96636764160,"available":96596299776,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":36890339},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:38:50.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:38:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:38:50.662 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-04-24T00:38:49.730029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":185758,"num_read_kb":1545653,"num_write":31374,"num_write_kb":1020589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8595,"num_bytes_recovered":360436542,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22942,"ondisk_log_size":22942,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":527540,"kb_used_data":27512,"kb_used_omap":17,"kb_used_meta":499886,"kb_avail":754447180,"statfs":{"total":773094113280,"available":772553912320,"internally_reserved":0,"allocated":28172288,"data_stored":15156078,"data_compressed":13970,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":17586,"internal_metadata":511884110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002013"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":5999,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062969+0000","last_change":"2024-04-24T00:38:18.803340+0000","last_active":"2024-04-24T00:38:46.062969+0000","last_peered":"2024-04-24T00:38:46.062969+0000","last_clean":"2024-04-24T00:38:46.062969+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:46.062969+0000","last_undegraded":"2024-04-24T00:38:46.062969+0000","last_fullsized":"2024-04-24T00:38:46.062969+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:16.140487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"264'1050","reported_seq":5685,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440757+0000","last_change":"2024-04-24T00:38:36.641262+0000","last_active":"2024-04-24T00:38:47.440757+0000","last_peered":"2024-04-24T00:38:47.440757+0000","last_clean":"2024-04-24T00:38:47.440757+0000","last_became_active":"2024-04-24T00:38:36.639573+0000","last_became_peered":"2024-04-24T00:38:36.639573+0000","last_unstale":"2024-04-24T00:38:47.440757+0000","last_undegraded":"2024-04-24T00:38:47.440757+0000","last_fullsized":"2024-04-24T00:38:47.440757+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:18:26.284735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080331599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"263'591","reported_seq":7547,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911549+0000","last_change":"2024-04-24T00:38:33.652375+0000","last_active":"2024-04-24T00:38:45.911549+0000","last_peered":"2024-04-24T00:38:45.911549+0000","last_clean":"2024-04-24T00:38:45.911549+0000","last_became_active":"2024-04-24T00:38:33.649639+0000","last_became_peered":"2024-04-24T00:38:33.649639+0000","last_unstale":"2024-04-24T00:38:45.911549+0000","last_undegraded":"2024-04-24T00:38:45.911549+0000","last_fullsized":"2024-04-24T00:38:45.911549+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:36.602084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"266'795","reported_seq":6337,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:38:35.634698+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:38:35.628937+0000","last_became_peered":"2024-04-24T00:38:35.628937+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:48.395425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5266,"num_read_kb":61368,"num_write":1008,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"264'498","reported_seq":4556,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911438+0000","last_change":"2024-04-24T00:38:18.764647+0000","last_active":"2024-04-24T00:38:45.911438+0000","last_peered":"2024-04-24T00:38:45.911438+0000","last_clean":"2024-04-24T00:38:45.911438+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:45.911438+0000","last_undegraded":"2024-04-24T00:38:45.911438+0000","last_fullsized":"2024-04-24T00:38:45.911438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:59.132716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"265'390","reported_seq":6310,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910619+0000","last_change":"2024-04-24T00:38:35.638402+0000","last_active":"2024-04-24T00:38:45.910619+0000","last_peered":"2024-04-24T00:38:45.910619+0000","last_clean":"2024-04-24T00:38:45.910619+0000","last_became_active":"2024-04-24T00:38:35.631522+0000","last_became_peered":"2024-04-24T00:38:35.631522+0000","last_unstale":"2024-04-24T00:38:45.910619+0000","last_undegraded":"2024-04-24T00:38:45.910619+0000","last_fullsized":"2024-04-24T00:38:45.910619+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:44.645527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001287217,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4509,"num_read_kb":37330,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":3953416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"265'712","reported_seq":5909,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909836+0000","last_change":"2024-04-24T00:38:18.779600+0000","last_active":"2024-04-24T00:38:45.909836+0000","last_peered":"2024-04-24T00:38:45.909836+0000","last_clean":"2024-04-24T00:38:45.909836+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:45.909836+0000","last_undegraded":"2024-04-24T00:38:45.909836+0000","last_fullsized":"2024-04-24T00:38:45.909836+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:58.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059221100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"263'551","reported_seq":4879,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440456+0000","last_change":"2024-04-24T00:38:36.632405+0000","last_active":"2024-04-24T00:38:47.440456+0000","last_peered":"2024-04-24T00:38:47.440456+0000","last_clean":"2024-04-24T00:38:47.440456+0000","last_became_active":"2024-04-24T00:38:36.631395+0000","last_became_peered":"2024-04-24T00:38:36.631395+0000","last_unstale":"2024-04-24T00:38:47.440456+0000","last_undegraded":"2024-04-24T00:38:47.440456+0000","last_fullsized":"2024-04-24T00:38:47.440456+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:35:30.536918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001058769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"263'339","reported_seq":4813,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063021+0000","last_change":"2024-04-24T00:38:18.803619+0000","last_active":"2024-04-24T00:38:46.063021+0000","last_peered":"2024-04-24T00:38:46.063021+0000","last_clean":"2024-04-24T00:38:46.063021+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:46.063021+0000","last_undegraded":"2024-04-24T00:38:46.063021+0000","last_fullsized":"2024-04-24T00:38:46.063021+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:53.921153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"266'751","reported_seq":6816,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440709+0000","last_change":"2024-04-24T00:38:18.756271+0000","last_active":"2024-04-24T00:38:47.440709+0000","last_peered":"2024-04-24T00:38:47.440709+0000","last_clean":"2024-04-24T00:38:47.440709+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:47.440709+0000","last_undegraded":"2024-04-24T00:38:47.440709+0000","last_fullsized":"2024-04-24T00:38:47.440709+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:36.855739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7650,"num_read_kb":50977,"num_write":1087,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"264'632","reported_seq":7275,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813710+0000","last_change":"2024-04-24T00:38:32.649992+0000","last_active":"2024-04-24T00:38:46.813710+0000","last_peered":"2024-04-24T00:38:46.813710+0000","last_clean":"2024-04-24T00:38:46.813710+0000","last_became_active":"2024-04-24T00:38:32.643949+0000","last_became_peered":"2024-04-24T00:38:32.643949+0000","last_unstale":"2024-04-24T00:38:46.813710+0000","last_undegraded":"2024-04-24T00:38:46.813710+0000","last_fullsized":"2024-04-24T00:38:46.813710+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:29:36.206394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"264'496","reported_seq":5383,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911030+0000","last_change":"2024-04-24T00:38:34.636361+0000","last_active":"2024-04-24T00:38:45.911030+0000","last_peered":"2024-04-24T00:38:45.911030+0000","last_clean":"2024-04-24T00:38:45.911030+0000","last_became_active":"2024-04-24T00:38:34.632609+0000","last_became_peered":"2024-04-24T00:38:34.632609+0000","last_unstale":"2024-04-24T00:38:45.911030+0000","last_undegraded":"2024-04-24T00:38:45.911030+0000","last_fullsized":"2024-04-24T00:38:45.911030+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:51:37.284131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010187200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":560,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107558+0000","last_change":"2024-04-24T00:38:30.696852+0000","last_active":"2024-04-24T00:38:46.107558+0000","last_peered":"2024-04-24T00:38:46.107558+0000","last_clean":"2024-04-24T00:38:46.107558+0000","last_became_active":"2024-04-24T00:38:30.696223+0000","last_became_peered":"2024-04-24T00:38:30.696223+0000","last_unstale":"2024-04-24T00:38:46.107558+0000","last_undegraded":"2024-04-24T00:38:46.107558+0000","last_fullsized":"2024-04-24T00:38:46.107558+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:08:35.422980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6523,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063089+0000","last_change":"2024-04-24T00:38:18.803861+0000","last_active":"2024-04-24T00:38:46.063089+0000","last_peered":"2024-04-24T00:38:46.063089+0000","last_clean":"2024-04-24T00:38:46.063089+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:46.063089+0000","last_undegraded":"2024-04-24T00:38:46.063089+0000","last_fullsized":"2024-04-24T00:38:46.063089+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:52.120677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910495+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:38:45.910495+0000","last_peered":"2024-04-24T00:38:45.910495+0000","last_clean":"2024-04-24T00:38:45.910495+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:45.910495+0000","last_undegraded":"2024-04-24T00:38:45.910495+0000","last_fullsized":"2024-04-24T00:38:45.910495+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4788,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440816+0000","last_change":"2024-04-24T00:38:36.639909+0000","last_active":"2024-04-24T00:38:47.440816+0000","last_peered":"2024-04-24T00:38:47.440816+0000","last_clean":"2024-04-24T00:38:47.440816+0000","last_became_active":"2024-04-24T00:38:36.637958+0000","last_became_peered":"2024-04-24T00:38:36.637958+0000","last_unstale":"2024-04-24T00:38:47.440816+0000","last_undegraded":"2024-04-24T00:38:47.440816+0000","last_fullsized":"2024-04-24T00:38:47.440816+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:23.662984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017453659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813411+0000","last_change":"2024-04-24T00:38:30.641161+0000","last_active":"2024-04-24T00:38:46.813411+0000","last_peered":"2024-04-24T00:38:46.813411+0000","last_clean":"2024-04-24T00:38:46.813411+0000","last_became_active":"2024-04-24T00:38:30.640288+0000","last_became_peered":"2024-04-24T00:38:30.640288+0000","last_unstale":"2024-04-24T00:38:46.813411+0000","last_undegraded":"2024-04-24T00:38:46.813411+0000","last_fullsized":"2024-04-24T00:38:46.813411+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:46:24.491722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4161,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910063+0000","last_change":"2024-04-24T00:38:18.780633+0000","last_active":"2024-04-24T00:38:45.910063+0000","last_peered":"2024-04-24T00:38:45.910063+0000","last_clean":"2024-04-24T00:38:45.910063+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:45.910063+0000","last_undegraded":"2024-04-24T00:38:45.910063+0000","last_fullsized":"2024-04-24T00:38:45.910063+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:46.783523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099788599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909769+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:38:45.909769+0000","last_peered":"2024-04-24T00:38:45.909769+0000","last_clean":"2024-04-24T00:38:45.909769+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:45.909769+0000","last_undegraded":"2024-04-24T00:38:45.909769+0000","last_fullsized":"2024-04-24T00:38:45.909769+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6292,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911342+0000","last_change":"2024-04-24T00:38:18.767450+0000","last_active":"2024-04-24T00:38:45.911342+0000","last_peered":"2024-04-24T00:38:45.911342+0000","last_clean":"2024-04-24T00:38:45.911342+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:45.911342+0000","last_undegraded":"2024-04-24T00:38:45.911342+0000","last_fullsized":"2024-04-24T00:38:45.911342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:09:34.691573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"266'580","reported_seq":6804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910527+0000","last_change":"2024-04-24T00:38:32.649496+0000","last_active":"2024-04-24T00:38:45.910527+0000","last_peered":"2024-04-24T00:38:45.910527+0000","last_clean":"2024-04-24T00:38:45.910527+0000","last_became_active":"2024-04-24T00:38:32.645508+0000","last_became_peered":"2024-04-24T00:38:32.645508+0000","last_unstale":"2024-04-24T00:38:45.910527+0000","last_undegraded":"2024-04-24T00:38:45.910527+0000","last_fullsized":"2024-04-24T00:38:45.910527+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:59.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910983+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:38:45.910983+0000","last_peered":"2024-04-24T00:38:45.910983+0000","last_clean":"2024-04-24T00:38:45.910983+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:45.910983+0000","last_undegraded":"2024-04-24T00:38:45.910983+0000","last_fullsized":"2024-04-24T00:38:45.910983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7975,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909664+0000","last_change":"2024-04-24T00:38:18.780209+0000","last_active":"2024-04-24T00:38:45.909664+0000","last_peered":"2024-04-24T00:38:45.909664+0000","last_clean":"2024-04-24T00:38:45.909664+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:45.909664+0000","last_undegraded":"2024-04-24T00:38:45.909664+0000","last_fullsized":"2024-04-24T00:38:45.909664+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:49.181426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":550,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370389+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:38:46.370389+0000","last_peered":"2024-04-24T00:38:46.370389+0000","last_clean":"2024-04-24T00:38:46.370389+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:46.370389+0000","last_undegraded":"2024-04-24T00:38:46.370389+0000","last_fullsized":"2024-04-24T00:38:46.370389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":5035,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063003+0000","last_change":"2024-04-24T00:38:34.632199+0000","last_active":"2024-04-24T00:38:46.063003+0000","last_peered":"2024-04-24T00:38:46.063003+0000","last_clean":"2024-04-24T00:38:46.063003+0000","last_became_active":"2024-04-24T00:38:34.629456+0000","last_became_peered":"2024-04-24T00:38:34.629456+0000","last_unstale":"2024-04-24T00:38:46.063003+0000","last_undegraded":"2024-04-24T00:38:46.063003+0000","last_fullsized":"2024-04-24T00:38:46.063003+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:05:27.920476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813416+0000","last_change":"2024-04-24T00:38:30.634583+0000","last_active":"2024-04-24T00:38:46.813416+0000","last_peered":"2024-04-24T00:38:46.813416+0000","last_clean":"2024-04-24T00:38:46.813416+0000","last_became_active":"2024-04-24T00:38:30.633606+0000","last_became_peered":"2024-04-24T00:38:30.633606+0000","last_unstale":"2024-04-24T00:38:46.813416+0000","last_undegraded":"2024-04-24T00:38:46.813416+0000","last_fullsized":"2024-04-24T00:38:46.813416+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:24:57.510774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2875,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440503+0000","last_change":"2024-04-24T00:38:36.633952+0000","last_active":"2024-04-24T00:38:47.440503+0000","last_peered":"2024-04-24T00:38:47.440503+0000","last_clean":"2024-04-24T00:38:47.440503+0000","last_became_active":"2024-04-24T00:38:36.632239+0000","last_became_peered":"2024-04-24T00:38:36.632239+0000","last_unstale":"2024-04-24T00:38:47.440503+0000","last_undegraded":"2024-04-24T00:38:47.440503+0000","last_fullsized":"2024-04-24T00:38:47.440503+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:50.888554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012896030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":506,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813566+0000","last_change":"2024-04-24T00:38:36.642364+0000","last_active":"2024-04-24T00:38:46.813566+0000","last_peered":"2024-04-24T00:38:46.813566+0000","last_clean":"2024-04-24T00:38:46.813566+0000","last_became_active":"2024-04-24T00:38:36.641100+0000","last_became_peered":"2024-04-24T00:38:36.641100+0000","last_unstale":"2024-04-24T00:38:46.813566+0000","last_undegraded":"2024-04-24T00:38:46.813566+0000","last_fullsized":"2024-04-24T00:38:46.813566+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:00.586831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6985,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440455+0000","last_change":"2024-04-24T00:38:18.755623+0000","last_active":"2024-04-24T00:38:47.440455+0000","last_peered":"2024-04-24T00:38:47.440455+0000","last_clean":"2024-04-24T00:38:47.440455+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:47.440455+0000","last_undegraded":"2024-04-24T00:38:47.440455+0000","last_fullsized":"2024-04-24T00:38:47.440455+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:44.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"265'987","reported_seq":5319,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910553+0000","last_change":"2024-04-24T00:38:33.656528+0000","last_active":"2024-04-24T00:38:45.910553+0000","last_peered":"2024-04-24T00:38:45.910553+0000","last_clean":"2024-04-24T00:38:45.910553+0000","last_became_active":"2024-04-24T00:38:33.650599+0000","last_became_peered":"2024-04-24T00:38:33.650599+0000","last_unstale":"2024-04-24T00:38:45.910553+0000","last_undegraded":"2024-04-24T00:38:45.910553+0000","last_fullsized":"2024-04-24T00:38:45.910553+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:19.834939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"263'1904","reported_seq":4804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107528+0000","last_change":"2024-04-24T00:38:34.638496+0000","last_active":"2024-04-24T00:38:46.107528+0000","last_peered":"2024-04-24T00:38:46.107528+0000","last_clean":"2024-04-24T00:38:46.107528+0000","last_became_active":"2024-04-24T00:38:34.634199+0000","last_became_peered":"2024-04-24T00:38:34.634199+0000","last_unstale":"2024-04-24T00:38:46.107528+0000","last_undegraded":"2024-04-24T00:38:46.107528+0000","last_fullsized":"2024-04-24T00:38:46.107528+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:57.455108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001715414,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"263'461","reported_seq":4303,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440544+0000","last_change":"2024-04-24T00:38:36.634741+0000","last_active":"2024-04-24T00:38:47.440544+0000","last_peered":"2024-04-24T00:38:47.440544+0000","last_clean":"2024-04-24T00:38:47.440544+0000","last_became_active":"2024-04-24T00:38:36.633468+0000","last_became_peered":"2024-04-24T00:38:36.633468+0000","last_unstale":"2024-04-24T00:38:47.440544+0000","last_undegraded":"2024-04-24T00:38:47.440544+0000","last_fullsized":"2024-04-24T00:38:47.440544+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:40.363226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002035177,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"266'831","reported_seq":10117,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911887+0000","last_change":"2024-04-24T00:38:18.766586+0000","last_active":"2024-04-24T00:38:45.911887+0000","last_peered":"2024-04-24T00:38:45.911887+0000","last_clean":"2024-04-24T00:38:45.911887+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:45.911887+0000","last_undegraded":"2024-04-24T00:38:45.911887+0000","last_fullsized":"2024-04-24T00:38:45.911887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:15.463491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10600,"num_read_kb":68795,"num_write":1180,"num_write_kb":35219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"263'426","reported_seq":2956,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813960+0000","last_change":"2024-04-24T00:38:35.637609+0000","last_active":"2024-04-24T00:38:46.813960+0000","last_peered":"2024-04-24T00:38:46.813960+0000","last_clean":"2024-04-24T00:38:46.813960+0000","last_became_active":"2024-04-24T00:38:35.632277+0000","last_became_peered":"2024-04-24T00:38:35.632277+0000","last_unstale":"2024-04-24T00:38:46.813960+0000","last_undegraded":"2024-04-24T00:38:46.813960+0000","last_fullsized":"2024-04-24T00:38:46.813960+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:03.305609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"266'671","reported_seq":8093,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814115+0000","last_change":"2024-04-24T00:38:37.642821+0000","last_active":"2024-04-24T00:38:46.814115+0000","last_peered":"2024-04-24T00:38:46.814115+0000","last_clean":"2024-04-24T00:38:46.814115+0000","last_became_active":"2024-04-24T00:38:37.638998+0000","last_became_peered":"2024-04-24T00:38:37.638998+0000","last_unstale":"2024-04-24T00:38:46.814115+0000","last_undegraded":"2024-04-24T00:38:46.814115+0000","last_fullsized":"2024-04-24T00:38:46.814115+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:10:52.647284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001218907,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":952,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"263'874","reported_seq":5902,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814247+0000","last_change":"2024-04-24T00:38:36.646741+0000","last_active":"2024-04-24T00:38:46.814247+0000","last_peered":"2024-04-24T00:38:46.814247+0000","last_clean":"2024-04-24T00:38:46.814247+0000","last_became_active":"2024-04-24T00:38:36.640462+0000","last_became_peered":"2024-04-24T00:38:36.640462+0000","last_unstale":"2024-04-24T00:38:46.814247+0000","last_undegraded":"2024-04-24T00:38:46.814247+0000","last_fullsized":"2024-04-24T00:38:46.814247+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:25.891296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"265'610","reported_seq":6427,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107457+0000","last_change":"2024-04-24T00:38:18.844983+0000","last_active":"2024-04-24T00:38:46.107457+0000","last_peered":"2024-04-24T00:38:46.107457+0000","last_clean":"2024-04-24T00:38:46.107457+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:46.107457+0000","last_undegraded":"2024-04-24T00:38:46.107457+0000","last_fullsized":"2024-04-24T00:38:46.107457+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.135502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"265'598","reported_seq":8477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813747+0000","last_change":"2024-04-24T00:38:37.645213+0000","last_active":"2024-04-24T00:38:46.813747+0000","last_peered":"2024-04-24T00:38:46.813747+0000","last_clean":"2024-04-24T00:38:46.813747+0000","last_became_active":"2024-04-24T00:38:37.638797+0000","last_became_peered":"2024-04-24T00:38:37.638797+0000","last_unstale":"2024-04-24T00:38:46.813747+0000","last_undegraded":"2024-04-24T00:38:46.813747+0000","last_fullsized":"2024-04-24T00:38:46.813747+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:20.508035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"265'1515","reported_seq":6441,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062859+0000","last_change":"2024-04-24T00:38:38.645480+0000","last_active":"2024-04-24T00:38:46.062859+0000","last_peered":"2024-04-24T00:38:46.062859+0000","last_clean":"2024-04-24T00:38:46.062859+0000","last_became_active":"2024-04-24T00:38:38.642317+0000","last_became_peered":"2024-04-24T00:38:38.642317+0000","last_unstale":"2024-04-24T00:38:46.062859+0000","last_undegraded":"2024-04-24T00:38:46.062859+0000","last_fullsized":"2024-04-24T00:38:46.062859+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:08.440907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"263'651","reported_seq":5104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813680+0000","last_change":"2024-04-24T00:38:32.655883+0000","last_active":"2024-04-24T00:38:46.813680+0000","last_peered":"2024-04-24T00:38:46.813680+0000","last_clean":"2024-04-24T00:38:46.813680+0000","last_became_active":"2024-04-24T00:38:32.650425+0000","last_became_peered":"2024-04-24T00:38:32.650425+0000","last_unstale":"2024-04-24T00:38:46.813680+0000","last_undegraded":"2024-04-24T00:38:46.813680+0000","last_fullsized":"2024-04-24T00:38:46.813680+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.249883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013270599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185642,"num_read_kb":1545553,"num_write":31215,"num_write_kb":1018173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":358075070,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},"log_size":22825,"ondisk_log_size":22825,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":2951840,"data_compressed":34925,"data_compressed_allocated":184320,"data_compressed_original":2949120,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56860,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314980,"statfs":{"total":96636764160,"available":96578539520,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66548,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305292,"statfs":{"total":96636764160,"available":96568619008,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74872,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296968,"statfs":{"total":96636764160,"available":96560095232,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92924,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94278916,"statfs":{"total":96636764160,"available":96541609984,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74864,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296976,"statfs":{"total":96636764160,"available":96560103424,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60976,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310864,"statfs":{"total":96636764160,"available":96574324736,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60980,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310860,"statfs":{"total":96636764160,"available":96574320640,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893165,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39516,"kb_used_data":3468,"kb_used_omap":6,"kb_used_meta":36025,"kb_avail":94332324,"statfs":{"total":96636764160,"available":96596299776,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":36890339},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:38:50.664 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T00:38:50.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:38:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:50.951 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:38:50.967 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-04-24T00:38:49.730029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":185758,"num_read_kb":1545653,"num_write":31374,"num_write_kb":1020589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8595,"num_bytes_recovered":360436542,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22942,"ondisk_log_size":22942,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":527540,"kb_used_data":27512,"kb_used_omap":17,"kb_used_meta":499886,"kb_avail":754447180,"statfs":{"total":773094113280,"available":772553912320,"internally_reserved":0,"allocated":28172288,"data_stored":15156078,"data_compressed":13970,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":17586,"internal_metadata":511884110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002013"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":5999,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062969+0000","last_change":"2024-04-24T00:38:18.803340+0000","last_active":"2024-04-24T00:38:46.062969+0000","last_peered":"2024-04-24T00:38:46.062969+0000","last_clean":"2024-04-24T00:38:46.062969+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:46.062969+0000","last_undegraded":"2024-04-24T00:38:46.062969+0000","last_fullsized":"2024-04-24T00:38:46.062969+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:09:16.140487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"264'1050","reported_seq":5685,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440757+0000","last_change":"2024-04-24T00:38:36.641262+0000","last_active":"2024-04-24T00:38:47.440757+0000","last_peered":"2024-04-24T00:38:47.440757+0000","last_clean":"2024-04-24T00:38:47.440757+0000","last_became_active":"2024-04-24T00:38:36.639573+0000","last_became_peered":"2024-04-24T00:38:36.639573+0000","last_unstale":"2024-04-24T00:38:47.440757+0000","last_undegraded":"2024-04-24T00:38:47.440757+0000","last_fullsized":"2024-04-24T00:38:47.440757+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:18:26.284735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080331599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"263'591","reported_seq":7547,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911549+0000","last_change":"2024-04-24T00:38:33.652375+0000","last_active":"2024-04-24T00:38:45.911549+0000","last_peered":"2024-04-24T00:38:45.911549+0000","last_clean":"2024-04-24T00:38:45.911549+0000","last_became_active":"2024-04-24T00:38:33.649639+0000","last_became_peered":"2024-04-24T00:38:33.649639+0000","last_unstale":"2024-04-24T00:38:45.911549+0000","last_undegraded":"2024-04-24T00:38:45.911549+0000","last_fullsized":"2024-04-24T00:38:45.911549+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:36.602084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"266'795","reported_seq":6337,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:38:35.634698+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:38:35.628937+0000","last_became_peered":"2024-04-24T00:38:35.628937+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:48.395425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5266,"num_read_kb":61368,"num_write":1008,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"264'498","reported_seq":4556,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911438+0000","last_change":"2024-04-24T00:38:18.764647+0000","last_active":"2024-04-24T00:38:45.911438+0000","last_peered":"2024-04-24T00:38:45.911438+0000","last_clean":"2024-04-24T00:38:45.911438+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:45.911438+0000","last_undegraded":"2024-04-24T00:38:45.911438+0000","last_fullsized":"2024-04-24T00:38:45.911438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:59.132716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"265'390","reported_seq":6310,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910619+0000","last_change":"2024-04-24T00:38:35.638402+0000","last_active":"2024-04-24T00:38:45.910619+0000","last_peered":"2024-04-24T00:38:45.910619+0000","last_clean":"2024-04-24T00:38:45.910619+0000","last_became_active":"2024-04-24T00:38:35.631522+0000","last_became_peered":"2024-04-24T00:38:35.631522+0000","last_unstale":"2024-04-24T00:38:45.910619+0000","last_undegraded":"2024-04-24T00:38:45.910619+0000","last_fullsized":"2024-04-24T00:38:45.910619+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:44.645527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001287217,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4509,"num_read_kb":37330,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":3953416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"265'712","reported_seq":5909,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909836+0000","last_change":"2024-04-24T00:38:18.779600+0000","last_active":"2024-04-24T00:38:45.909836+0000","last_peered":"2024-04-24T00:38:45.909836+0000","last_clean":"2024-04-24T00:38:45.909836+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:45.909836+0000","last_undegraded":"2024-04-24T00:38:45.909836+0000","last_fullsized":"2024-04-24T00:38:45.909836+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:58.934641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059221100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"263'551","reported_seq":4879,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440456+0000","last_change":"2024-04-24T00:38:36.632405+0000","last_active":"2024-04-24T00:38:47.440456+0000","last_peered":"2024-04-24T00:38:47.440456+0000","last_clean":"2024-04-24T00:38:47.440456+0000","last_became_active":"2024-04-24T00:38:36.631395+0000","last_became_peered":"2024-04-24T00:38:36.631395+0000","last_unstale":"2024-04-24T00:38:47.440456+0000","last_undegraded":"2024-04-24T00:38:47.440456+0000","last_fullsized":"2024-04-24T00:38:47.440456+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:35:30.536918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001058769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"263'339","reported_seq":4813,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063021+0000","last_change":"2024-04-24T00:38:18.803619+0000","last_active":"2024-04-24T00:38:46.063021+0000","last_peered":"2024-04-24T00:38:46.063021+0000","last_clean":"2024-04-24T00:38:46.063021+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:46.063021+0000","last_undegraded":"2024-04-24T00:38:46.063021+0000","last_fullsized":"2024-04-24T00:38:46.063021+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:53.921153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"266'751","reported_seq":6816,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440709+0000","last_change":"2024-04-24T00:38:18.756271+0000","last_active":"2024-04-24T00:38:47.440709+0000","last_peered":"2024-04-24T00:38:47.440709+0000","last_clean":"2024-04-24T00:38:47.440709+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:47.440709+0000","last_undegraded":"2024-04-24T00:38:47.440709+0000","last_fullsized":"2024-04-24T00:38:47.440709+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:36.855739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7650,"num_read_kb":50977,"num_write":1087,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"264'632","reported_seq":7275,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813710+0000","last_change":"2024-04-24T00:38:32.649992+0000","last_active":"2024-04-24T00:38:46.813710+0000","last_peered":"2024-04-24T00:38:46.813710+0000","last_clean":"2024-04-24T00:38:46.813710+0000","last_became_active":"2024-04-24T00:38:32.643949+0000","last_became_peered":"2024-04-24T00:38:32.643949+0000","last_unstale":"2024-04-24T00:38:46.813710+0000","last_undegraded":"2024-04-24T00:38:46.813710+0000","last_fullsized":"2024-04-24T00:38:46.813710+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:29:36.206394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"264'496","reported_seq":5383,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911030+0000","last_change":"2024-04-24T00:38:34.636361+0000","last_active":"2024-04-24T00:38:45.911030+0000","last_peered":"2024-04-24T00:38:45.911030+0000","last_clean":"2024-04-24T00:38:45.911030+0000","last_became_active":"2024-04-24T00:38:34.632609+0000","last_became_peered":"2024-04-24T00:38:34.632609+0000","last_unstale":"2024-04-24T00:38:45.911030+0000","last_undegraded":"2024-04-24T00:38:45.911030+0000","last_fullsized":"2024-04-24T00:38:45.911030+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:51:37.284131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010187200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":560,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107558+0000","last_change":"2024-04-24T00:38:30.696852+0000","last_active":"2024-04-24T00:38:46.107558+0000","last_peered":"2024-04-24T00:38:46.107558+0000","last_clean":"2024-04-24T00:38:46.107558+0000","last_became_active":"2024-04-24T00:38:30.696223+0000","last_became_peered":"2024-04-24T00:38:30.696223+0000","last_unstale":"2024-04-24T00:38:46.107558+0000","last_undegraded":"2024-04-24T00:38:46.107558+0000","last_fullsized":"2024-04-24T00:38:46.107558+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:08:35.422980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6523,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063089+0000","last_change":"2024-04-24T00:38:18.803861+0000","last_active":"2024-04-24T00:38:46.063089+0000","last_peered":"2024-04-24T00:38:46.063089+0000","last_clean":"2024-04-24T00:38:46.063089+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:46.063089+0000","last_undegraded":"2024-04-24T00:38:46.063089+0000","last_fullsized":"2024-04-24T00:38:46.063089+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:52.120677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910495+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:38:45.910495+0000","last_peered":"2024-04-24T00:38:45.910495+0000","last_clean":"2024-04-24T00:38:45.910495+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:45.910495+0000","last_undegraded":"2024-04-24T00:38:45.910495+0000","last_fullsized":"2024-04-24T00:38:45.910495+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:25:08.481177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4788,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440816+0000","last_change":"2024-04-24T00:38:36.639909+0000","last_active":"2024-04-24T00:38:47.440816+0000","last_peered":"2024-04-24T00:38:47.440816+0000","last_clean":"2024-04-24T00:38:47.440816+0000","last_became_active":"2024-04-24T00:38:36.637958+0000","last_became_peered":"2024-04-24T00:38:36.637958+0000","last_unstale":"2024-04-24T00:38:47.440816+0000","last_undegraded":"2024-04-24T00:38:47.440816+0000","last_fullsized":"2024-04-24T00:38:47.440816+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:23.662984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017453659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813411+0000","last_change":"2024-04-24T00:38:30.641161+0000","last_active":"2024-04-24T00:38:46.813411+0000","last_peered":"2024-04-24T00:38:46.813411+0000","last_clean":"2024-04-24T00:38:46.813411+0000","last_became_active":"2024-04-24T00:38:30.640288+0000","last_became_peered":"2024-04-24T00:38:30.640288+0000","last_unstale":"2024-04-24T00:38:46.813411+0000","last_undegraded":"2024-04-24T00:38:46.813411+0000","last_fullsized":"2024-04-24T00:38:46.813411+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:46:24.491722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4161,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910063+0000","last_change":"2024-04-24T00:38:18.780633+0000","last_active":"2024-04-24T00:38:45.910063+0000","last_peered":"2024-04-24T00:38:45.910063+0000","last_clean":"2024-04-24T00:38:45.910063+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:45.910063+0000","last_undegraded":"2024-04-24T00:38:45.910063+0000","last_fullsized":"2024-04-24T00:38:45.910063+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:46.783523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099788599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":115,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909769+0000","last_change":"2024-04-24T00:37:48.620612+0000","last_active":"2024-04-24T00:38:45.909769+0000","last_peered":"2024-04-24T00:38:45.909769+0000","last_clean":"2024-04-24T00:38:45.909769+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:45.909769+0000","last_undegraded":"2024-04-24T00:38:45.909769+0000","last_fullsized":"2024-04-24T00:38:45.909769+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:45.191535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6292,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911342+0000","last_change":"2024-04-24T00:38:18.767450+0000","last_active":"2024-04-24T00:38:45.911342+0000","last_peered":"2024-04-24T00:38:45.911342+0000","last_clean":"2024-04-24T00:38:45.911342+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:45.911342+0000","last_undegraded":"2024-04-24T00:38:45.911342+0000","last_fullsized":"2024-04-24T00:38:45.911342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:09:34.691573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"266'580","reported_seq":6804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910527+0000","last_change":"2024-04-24T00:38:32.649496+0000","last_active":"2024-04-24T00:38:45.910527+0000","last_peered":"2024-04-24T00:38:45.910527+0000","last_clean":"2024-04-24T00:38:45.910527+0000","last_became_active":"2024-04-24T00:38:32.645508+0000","last_became_peered":"2024-04-24T00:38:32.645508+0000","last_unstale":"2024-04-24T00:38:45.910527+0000","last_undegraded":"2024-04-24T00:38:45.910527+0000","last_fullsized":"2024-04-24T00:38:45.910527+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:59.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910983+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:38:45.910983+0000","last_peered":"2024-04-24T00:38:45.910983+0000","last_clean":"2024-04-24T00:38:45.910983+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:45.910983+0000","last_undegraded":"2024-04-24T00:38:45.910983+0000","last_fullsized":"2024-04-24T00:38:45.910983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7975,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909664+0000","last_change":"2024-04-24T00:38:18.780209+0000","last_active":"2024-04-24T00:38:45.909664+0000","last_peered":"2024-04-24T00:38:45.909664+0000","last_clean":"2024-04-24T00:38:45.909664+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:45.909664+0000","last_undegraded":"2024-04-24T00:38:45.909664+0000","last_fullsized":"2024-04-24T00:38:45.909664+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:34:49.181426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":550,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370389+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:38:46.370389+0000","last_peered":"2024-04-24T00:38:46.370389+0000","last_clean":"2024-04-24T00:38:46.370389+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:46.370389+0000","last_undegraded":"2024-04-24T00:38:46.370389+0000","last_fullsized":"2024-04-24T00:38:46.370389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:42:40.879708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":5035,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063003+0000","last_change":"2024-04-24T00:38:34.632199+0000","last_active":"2024-04-24T00:38:46.063003+0000","last_peered":"2024-04-24T00:38:46.063003+0000","last_clean":"2024-04-24T00:38:46.063003+0000","last_became_active":"2024-04-24T00:38:34.629456+0000","last_became_peered":"2024-04-24T00:38:34.629456+0000","last_unstale":"2024-04-24T00:38:46.063003+0000","last_undegraded":"2024-04-24T00:38:46.063003+0000","last_fullsized":"2024-04-24T00:38:46.063003+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:05:27.920476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813416+0000","last_change":"2024-04-24T00:38:30.634583+0000","last_active":"2024-04-24T00:38:46.813416+0000","last_peered":"2024-04-24T00:38:46.813416+0000","last_clean":"2024-04-24T00:38:46.813416+0000","last_became_active":"2024-04-24T00:38:30.633606+0000","last_became_peered":"2024-04-24T00:38:30.633606+0000","last_unstale":"2024-04-24T00:38:46.813416+0000","last_undegraded":"2024-04-24T00:38:46.813416+0000","last_fullsized":"2024-04-24T00:38:46.813416+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:24:57.510774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2875,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440503+0000","last_change":"2024-04-24T00:38:36.633952+0000","last_active":"2024-04-24T00:38:47.440503+0000","last_peered":"2024-04-24T00:38:47.440503+0000","last_clean":"2024-04-24T00:38:47.440503+0000","last_became_active":"2024-04-24T00:38:36.632239+0000","last_became_peered":"2024-04-24T00:38:36.632239+0000","last_unstale":"2024-04-24T00:38:47.440503+0000","last_undegraded":"2024-04-24T00:38:47.440503+0000","last_fullsized":"2024-04-24T00:38:47.440503+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:50.888554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012896030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":506,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813566+0000","last_change":"2024-04-24T00:38:36.642364+0000","last_active":"2024-04-24T00:38:46.813566+0000","last_peered":"2024-04-24T00:38:46.813566+0000","last_clean":"2024-04-24T00:38:46.813566+0000","last_became_active":"2024-04-24T00:38:36.641100+0000","last_became_peered":"2024-04-24T00:38:36.641100+0000","last_unstale":"2024-04-24T00:38:46.813566+0000","last_undegraded":"2024-04-24T00:38:46.813566+0000","last_fullsized":"2024-04-24T00:38:46.813566+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:00.586831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6985,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440455+0000","last_change":"2024-04-24T00:38:18.755623+0000","last_active":"2024-04-24T00:38:47.440455+0000","last_peered":"2024-04-24T00:38:47.440455+0000","last_clean":"2024-04-24T00:38:47.440455+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:47.440455+0000","last_undegraded":"2024-04-24T00:38:47.440455+0000","last_fullsized":"2024-04-24T00:38:47.440455+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:44.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"265'987","reported_seq":5319,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910553+0000","last_change":"2024-04-24T00:38:33.656528+0000","last_active":"2024-04-24T00:38:45.910553+0000","last_peered":"2024-04-24T00:38:45.910553+0000","last_clean":"2024-04-24T00:38:45.910553+0000","last_became_active":"2024-04-24T00:38:33.650599+0000","last_became_peered":"2024-04-24T00:38:33.650599+0000","last_unstale":"2024-04-24T00:38:45.910553+0000","last_undegraded":"2024-04-24T00:38:45.910553+0000","last_fullsized":"2024-04-24T00:38:45.910553+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:19.834939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"263'1904","reported_seq":4804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107528+0000","last_change":"2024-04-24T00:38:34.638496+0000","last_active":"2024-04-24T00:38:46.107528+0000","last_peered":"2024-04-24T00:38:46.107528+0000","last_clean":"2024-04-24T00:38:46.107528+0000","last_became_active":"2024-04-24T00:38:34.634199+0000","last_became_peered":"2024-04-24T00:38:34.634199+0000","last_unstale":"2024-04-24T00:38:46.107528+0000","last_undegraded":"2024-04-24T00:38:46.107528+0000","last_fullsized":"2024-04-24T00:38:46.107528+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:57.455108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001715414,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"263'461","reported_seq":4303,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440544+0000","last_change":"2024-04-24T00:38:36.634741+0000","last_active":"2024-04-24T00:38:47.440544+0000","last_peered":"2024-04-24T00:38:47.440544+0000","last_clean":"2024-04-24T00:38:47.440544+0000","last_became_active":"2024-04-24T00:38:36.633468+0000","last_became_peered":"2024-04-24T00:38:36.633468+0000","last_unstale":"2024-04-24T00:38:47.440544+0000","last_undegraded":"2024-04-24T00:38:47.440544+0000","last_fullsized":"2024-04-24T00:38:47.440544+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:40.363226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002035177,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"266'831","reported_seq":10117,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911887+0000","last_change":"2024-04-24T00:38:18.766586+0000","last_active":"2024-04-24T00:38:45.911887+0000","last_peered":"2024-04-24T00:38:45.911887+0000","last_clean":"2024-04-24T00:38:45.911887+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:45.911887+0000","last_undegraded":"2024-04-24T00:38:45.911887+0000","last_fullsized":"2024-04-24T00:38:45.911887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:15.463491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10600,"num_read_kb":68795,"num_write":1180,"num_write_kb":35219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"263'426","reported_seq":2956,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813960+0000","last_change":"2024-04-24T00:38:35.637609+0000","last_active":"2024-04-24T00:38:46.813960+0000","last_peered":"2024-04-24T00:38:46.813960+0000","last_clean":"2024-04-24T00:38:46.813960+0000","last_became_active":"2024-04-24T00:38:35.632277+0000","last_became_peered":"2024-04-24T00:38:35.632277+0000","last_unstale":"2024-04-24T00:38:46.813960+0000","last_undegraded":"2024-04-24T00:38:46.813960+0000","last_fullsized":"2024-04-24T00:38:46.813960+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:03.305609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"266'671","reported_seq":8093,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814115+0000","last_change":"2024-04-24T00:38:37.642821+0000","last_active":"2024-04-24T00:38:46.814115+0000","last_peered":"2024-04-24T00:38:46.814115+0000","last_clean":"2024-04-24T00:38:46.814115+0000","last_became_active":"2024-04-24T00:38:37.638998+0000","last_became_peered":"2024-04-24T00:38:37.638998+0000","last_unstale":"2024-04-24T00:38:46.814115+0000","last_undegraded":"2024-04-24T00:38:46.814115+0000","last_fullsized":"2024-04-24T00:38:46.814115+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:10:52.647284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001218907,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":952,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"263'874","reported_seq":5902,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814247+0000","last_change":"2024-04-24T00:38:36.646741+0000","last_active":"2024-04-24T00:38:46.814247+0000","last_peered":"2024-04-24T00:38:46.814247+0000","last_clean":"2024-04-24T00:38:46.814247+0000","last_became_active":"2024-04-24T00:38:36.640462+0000","last_became_peered":"2024-04-24T00:38:36.640462+0000","last_unstale":"2024-04-24T00:38:46.814247+0000","last_undegraded":"2024-04-24T00:38:46.814247+0000","last_fullsized":"2024-04-24T00:38:46.814247+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:25.891296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"265'610","reported_seq":6427,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107457+0000","last_change":"2024-04-24T00:38:18.844983+0000","last_active":"2024-04-24T00:38:46.107457+0000","last_peered":"2024-04-24T00:38:46.107457+0000","last_clean":"2024-04-24T00:38:46.107457+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:46.107457+0000","last_undegraded":"2024-04-24T00:38:46.107457+0000","last_fullsized":"2024-04-24T00:38:46.107457+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.135502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"265'598","reported_seq":8477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813747+0000","last_change":"2024-04-24T00:38:37.645213+0000","last_active":"2024-04-24T00:38:46.813747+0000","last_peered":"2024-04-24T00:38:46.813747+0000","last_clean":"2024-04-24T00:38:46.813747+0000","last_became_active":"2024-04-24T00:38:37.638797+0000","last_became_peered":"2024-04-24T00:38:37.638797+0000","last_unstale":"2024-04-24T00:38:46.813747+0000","last_undegraded":"2024-04-24T00:38:46.813747+0000","last_fullsized":"2024-04-24T00:38:46.813747+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:20.508035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"265'1515","reported_seq":6441,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.062859+0000","last_change":"2024-04-24T00:38:38.645480+0000","last_active":"2024-04-24T00:38:46.062859+0000","last_peered":"2024-04-24T00:38:46.062859+0000","last_clean":"2024-04-24T00:38:46.062859+0000","last_became_active":"2024-04-24T00:38:38.642317+0000","last_became_peered":"2024-04-24T00:38:38.642317+0000","last_unstale":"2024-04-24T00:38:46.062859+0000","last_undegraded":"2024-04-24T00:38:46.062859+0000","last_fullsized":"2024-04-24T00:38:46.062859+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:08.440907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"263'651","reported_seq":5104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813680+0000","last_change":"2024-04-24T00:38:32.655883+0000","last_active":"2024-04-24T00:38:46.813680+0000","last_peered":"2024-04-24T00:38:46.813680+0000","last_clean":"2024-04-24T00:38:46.813680+0000","last_became_active":"2024-04-24T00:38:32.650425+0000","last_became_peered":"2024-04-24T00:38:32.650425+0000","last_unstale":"2024-04-24T00:38:46.813680+0000","last_undegraded":"2024-04-24T00:38:46.813680+0000","last_fullsized":"2024-04-24T00:38:46.813680+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.249883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013270599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185642,"num_read_kb":1545553,"num_write":31215,"num_write_kb":1018173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":358075070,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},"log_size":22825,"ondisk_log_size":22825,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":2951840,"data_compressed":34925,"data_compressed_allocated":184320,"data_compressed_original":2949120,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574916,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56860,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314980,"statfs":{"total":96636764160,"available":96578539520,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66548,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305292,"statfs":{"total":96636764160,"available":96568619008,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74872,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296968,"statfs":{"total":96636764160,"available":96560095232,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92924,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94278916,"statfs":{"total":96636764160,"available":96541609984,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74864,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296976,"statfs":{"total":96636764160,"available":96560103424,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60976,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310864,"statfs":{"total":96636764160,"available":96574324736,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60980,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310860,"statfs":{"total":96636764160,"available":96574320640,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893165,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39516,"kb_used_data":3468,"kb_used_omap":6,"kb_used_meta":36025,"kb_avail":94332324,"statfs":{"total":96636764160,"available":96596299776,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":36890339},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:38:50.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T00:38:51.267 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:51.267 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":289,"fsid":"6a70d87d-4453-47be-af2a-5590f9851254","created":"2024-04-24T00:33:24.857711+0000","modified":"2024-04-24T00:38:45.901888+0000","last_up_change":"2024-04-24T00:35:14.574643+0000","last_in_change":"2024-04-24T00:38:29.604271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T00:33:39.734814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T00:33:43.786311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-04-24T00:33:51.288702+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":85,"snap_epoch":266,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c80b3f4-99ee-4725-90a8-920116161bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":74,"up_from":95,"up_thru":281,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6817","nonce":218259947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6819","nonce":218259947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6823","nonce":218259947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":218259947},{"type":"v1","addr":"172.21.15.64:6821","nonce":218259947}]},"public_addr":"172.21.15.64:6817/218259947","cluster_addr":"172.21.15.64:6819/218259947","heartbeat_back_addr":"172.21.15.64:6823/218259947","heartbeat_front_addr":"172.21.15.64:6821/218259947","state":["exists","up"]},{"osd":1,"uuid":"20e1331a-f8a2-48fd-b2b8-b7a92f21d78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6825","nonce":637720411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6827","nonce":637720411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6831","nonce":637720411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":637720411},{"type":"v1","addr":"172.21.15.64:6829","nonce":637720411}]},"public_addr":"172.21.15.64:6825/637720411","cluster_addr":"172.21.15.64:6827/637720411","heartbeat_back_addr":"172.21.15.64:6831/637720411","heartbeat_front_addr":"172.21.15.64:6829/637720411","state":["exists","up"]},{"osd":2,"uuid":"a51557a0-9634-4a20-82b0-b54cdd36b166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6809","nonce":820024667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6811","nonce":820024667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6815","nonce":820024667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":820024667},{"type":"v1","addr":"172.21.15.64:6813","nonce":820024667}]},"public_addr":"172.21.15.64:6809/820024667","cluster_addr":"172.21.15.64:6811/820024667","heartbeat_back_addr":"172.21.15.64:6815/820024667","heartbeat_front_addr":"172.21.15.64:6813/820024667","state":["exists","up"]},{"osd":3,"uuid":"aafffa79-f532-4441-b1d6-f6441c5eebf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":279,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6801","nonce":1974534323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6803","nonce":1974534323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6807","nonce":1974534323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1974534323},{"type":"v1","addr":"172.21.15.64:6805","nonce":1974534323}]},"public_addr":"172.21.15.64:6801/1974534323","cluster_addr":"172.21.15.64:6803/1974534323","heartbeat_back_addr":"172.21.15.64:6807/1974534323","heartbeat_front_addr":"172.21.15.64:6805/1974534323","state":["exists","up"]},{"osd":4,"uuid":"78f05556-598e-4be2-b682-e3f58863eb82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":278,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6818","nonce":1642543604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6821","nonce":1642543604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6828","nonce":1642543604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6823","nonce":1642543604},{"type":"v1","addr":"172.21.15.167:6824","nonce":1642543604}]},"public_addr":"172.21.15.167:6818/1642543604","cluster_addr":"172.21.15.167:6821/1642543604","heartbeat_back_addr":"172.21.15.167:6828/1642543604","heartbeat_front_addr":"172.21.15.167:6824/1642543604","state":["exists","up"]},{"osd":5,"uuid":"65f7d2ed-23b9-4a3c-bbfb-3a88d181f8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":278,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6809","nonce":172726194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6811","nonce":172726194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6815","nonce":172726194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":172726194},{"type":"v1","addr":"172.21.15.167:6813","nonce":172726194}]},"public_addr":"172.21.15.167:6809/172726194","cluster_addr":"172.21.15.167:6811/172726194","heartbeat_back_addr":"172.21.15.167:6815/172726194","heartbeat_front_addr":"172.21.15.167:6813/172726194","state":["exists","up"]},{"osd":6,"uuid":"6dad12bf-94de-482a-a402-49e67d5e5dc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6801","nonce":2195746987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6803","nonce":2195746987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6807","nonce":2195746987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":2195746987},{"type":"v1","addr":"172.21.15.167:6805","nonce":2195746987}]},"public_addr":"172.21.15.167:6801/2195746987","cluster_addr":"172.21.15.167:6803/2195746987","heartbeat_back_addr":"172.21.15.167:6807/2195746987","heartbeat_front_addr":"172.21.15.167:6805/2195746987","state":["exists","up"]},{"osd":7,"uuid":"e6734cfe-3c24-4ddd-8ef0-5ed450efe772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":280,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6817","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6819","nonce":954791674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6825","nonce":954791674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6831","nonce":954791674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6827","nonce":954791674},{"type":"v1","addr":"172.21.15.167:6829","nonce":954791674}]},"public_addr":"172.21.15.167:6819/954791674","cluster_addr":"172.21.15.167:6825/954791674","heartbeat_back_addr":"172.21.15.167:6831/954791674","heartbeat_front_addr":"172.21.15.167:6829/954791674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.509233+0000","dead_epoch":75},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.465696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.405177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.269202+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.359180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.351834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.490127+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T00:33:33.484842+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.16","osds":[3,2]},{"pgid":"3.1b","osds":[5,6]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T00:38:52.287 INFO:tasks.ceph:Scrubbing osd.0 2024-04-24T00:38:52.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T00:38:52.454 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:52.454 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T00:38:52.454 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:52.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-24T00:38:52.763 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T00:38:52.782 INFO:tasks.ceph:Scrubbing osd.1 2024-04-24T00:38:52.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T00:38:52.930 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:52.930 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T00:38:52.930 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:52.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-24T00:38:53.237 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T00:38:53.256 INFO:tasks.ceph:Scrubbing osd.2 2024-04-24T00:38:53.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T00:38:53.409 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:53.409 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T00:38:53.410 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:53.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-24T00:38:53.720 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T00:38:53.740 INFO:tasks.ceph:Scrubbing osd.3 2024-04-24T00:38:53.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-24T00:38:53.897 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:53.897 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T00:38:53.897 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:53.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-24T00:38:54.207 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 3 to deep-scrub 2024-04-24T00:38:54.227 INFO:tasks.ceph:Scrubbing osd.4 2024-04-24T00:38:54.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-24T00:38:54.386 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:54.386 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T00:38:54.386 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:54.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-24T00:38:54.699 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 4 to deep-scrub 2024-04-24T00:38:54.717 INFO:tasks.ceph:Scrubbing osd.5 2024-04-24T00:38:54.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-24T00:38:54.871 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:54.871 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T00:38:54.871 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:54.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-24T00:38:55.179 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 5 to deep-scrub 2024-04-24T00:38:55.199 INFO:tasks.ceph:Scrubbing osd.6 2024-04-24T00:38:55.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-24T00:38:55.349 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:55.349 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T00:38:55.349 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:55.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-24T00:38:55.656 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 6 to deep-scrub 2024-04-24T00:38:55.674 INFO:tasks.ceph:Scrubbing osd.7 2024-04-24T00:38:55.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-24T00:38:55.828 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-24T00:38:55.828 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T00:38:55.829 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-24T00:38:55.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-24T00:38:56.132 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 7 to deep-scrub 2024-04-24T00:38:56.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:38:56.455 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:38:56.456 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:38:56.475 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-04-24T00:38:55.731794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":185758,"num_read_kb":1545653,"num_write":31374,"num_write_kb":1020589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8595,"num_bytes_recovered":360436542,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22942,"ondisk_log_size":22942,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":527532,"kb_used_data":27512,"kb_used_omap":17,"kb_used_meta":499886,"kb_avail":754447188,"statfs":{"total":773094113280,"available":772553920512,"internally_reserved":0,"allocated":28172288,"data_stored":15156078,"data_compressed":13970,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":17586,"internal_metadata":511884110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003200"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":6008,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:54.535891+0000","last_change":"2024-04-24T00:38:54.535917+0000","last_active":"2024-04-24T00:38:54.535891+0000","last_peered":"2024-04-24T00:38:54.535891+0000","last_clean":"2024-04-24T00:38:54.535891+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:54.535891+0000","last_undegraded":"2024-04-24T00:38:54.535891+0000","last_fullsized":"2024-04-24T00:38:54.535891+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'878","last_scrub_stamp":"2024-04-24T00:38:54.535792+0000","last_deep_scrub":"266'878","last_deep_scrub_stamp":"2024-04-24T00:38:54.535792+0000","last_clean_scrub_stamp":"2024-04-24T00:38:54.535792+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:51:37.763435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"264'1050","reported_seq":5685,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440757+0000","last_change":"2024-04-24T00:38:36.641262+0000","last_active":"2024-04-24T00:38:47.440757+0000","last_peered":"2024-04-24T00:38:47.440757+0000","last_clean":"2024-04-24T00:38:47.440757+0000","last_became_active":"2024-04-24T00:38:36.639573+0000","last_became_peered":"2024-04-24T00:38:36.639573+0000","last_unstale":"2024-04-24T00:38:47.440757+0000","last_undegraded":"2024-04-24T00:38:47.440757+0000","last_fullsized":"2024-04-24T00:38:47.440757+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:18:26.284735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080331599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"263'591","reported_seq":7547,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911549+0000","last_change":"2024-04-24T00:38:33.652375+0000","last_active":"2024-04-24T00:38:45.911549+0000","last_peered":"2024-04-24T00:38:45.911549+0000","last_clean":"2024-04-24T00:38:45.911549+0000","last_became_active":"2024-04-24T00:38:33.649639+0000","last_became_peered":"2024-04-24T00:38:33.649639+0000","last_unstale":"2024-04-24T00:38:45.911549+0000","last_undegraded":"2024-04-24T00:38:45.911549+0000","last_fullsized":"2024-04-24T00:38:45.911549+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:36.602084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"266'795","reported_seq":6337,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:38:35.634698+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:38:35.628937+0000","last_became_peered":"2024-04-24T00:38:35.628937+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:48.395425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5266,"num_read_kb":61368,"num_write":1008,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"264'498","reported_seq":4556,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911438+0000","last_change":"2024-04-24T00:38:18.764647+0000","last_active":"2024-04-24T00:38:45.911438+0000","last_peered":"2024-04-24T00:38:45.911438+0000","last_clean":"2024-04-24T00:38:45.911438+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:45.911438+0000","last_undegraded":"2024-04-24T00:38:45.911438+0000","last_fullsized":"2024-04-24T00:38:45.911438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:59.132716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"265'390","reported_seq":6310,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910619+0000","last_change":"2024-04-24T00:38:35.638402+0000","last_active":"2024-04-24T00:38:45.910619+0000","last_peered":"2024-04-24T00:38:45.910619+0000","last_clean":"2024-04-24T00:38:45.910619+0000","last_became_active":"2024-04-24T00:38:35.631522+0000","last_became_peered":"2024-04-24T00:38:35.631522+0000","last_unstale":"2024-04-24T00:38:45.910619+0000","last_undegraded":"2024-04-24T00:38:45.910619+0000","last_fullsized":"2024-04-24T00:38:45.910619+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:37:44.645527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001287217,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4509,"num_read_kb":37330,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":3953416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"265'712","reported_seq":5910,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909951+0000","last_change":"2024-04-24T00:38:18.779600+0000","last_active":"2024-04-24T00:38:45.909951+0000","last_peered":"2024-04-24T00:38:45.909951+0000","last_clean":"2024-04-24T00:38:45.909951+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:45.909951+0000","last_undegraded":"2024-04-24T00:38:45.909951+0000","last_fullsized":"2024-04-24T00:38:45.909951+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059221100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"263'551","reported_seq":4879,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440456+0000","last_change":"2024-04-24T00:38:36.632405+0000","last_active":"2024-04-24T00:38:47.440456+0000","last_peered":"2024-04-24T00:38:47.440456+0000","last_clean":"2024-04-24T00:38:47.440456+0000","last_became_active":"2024-04-24T00:38:36.631395+0000","last_became_peered":"2024-04-24T00:38:36.631395+0000","last_unstale":"2024-04-24T00:38:47.440456+0000","last_undegraded":"2024-04-24T00:38:47.440456+0000","last_fullsized":"2024-04-24T00:38:47.440456+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:35:30.536918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001058769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"263'339","reported_seq":4822,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:53.508792+0000","last_change":"2024-04-24T00:38:53.508850+0000","last_active":"2024-04-24T00:38:53.508792+0000","last_peered":"2024-04-24T00:38:53.508792+0000","last_clean":"2024-04-24T00:38:53.508792+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:53.508792+0000","last_undegraded":"2024-04-24T00:38:53.508792+0000","last_fullsized":"2024-04-24T00:38:53.508792+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'339","last_scrub_stamp":"2024-04-24T00:38:53.508594+0000","last_deep_scrub":"263'339","last_deep_scrub_stamp":"2024-04-24T00:38:53.508594+0000","last_clean_scrub_stamp":"2024-04-24T00:38:53.508594+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:05:28.524601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"266'751","reported_seq":6816,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440709+0000","last_change":"2024-04-24T00:38:18.756271+0000","last_active":"2024-04-24T00:38:47.440709+0000","last_peered":"2024-04-24T00:38:47.440709+0000","last_clean":"2024-04-24T00:38:47.440709+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:47.440709+0000","last_undegraded":"2024-04-24T00:38:47.440709+0000","last_fullsized":"2024-04-24T00:38:47.440709+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:36.855739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7650,"num_read_kb":50977,"num_write":1087,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"264'632","reported_seq":7275,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813710+0000","last_change":"2024-04-24T00:38:32.649992+0000","last_active":"2024-04-24T00:38:46.813710+0000","last_peered":"2024-04-24T00:38:46.813710+0000","last_clean":"2024-04-24T00:38:46.813710+0000","last_became_active":"2024-04-24T00:38:32.643949+0000","last_became_peered":"2024-04-24T00:38:32.643949+0000","last_unstale":"2024-04-24T00:38:46.813710+0000","last_undegraded":"2024-04-24T00:38:46.813710+0000","last_fullsized":"2024-04-24T00:38:46.813710+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:29:36.206394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"264'496","reported_seq":5383,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911030+0000","last_change":"2024-04-24T00:38:34.636361+0000","last_active":"2024-04-24T00:38:45.911030+0000","last_peered":"2024-04-24T00:38:45.911030+0000","last_clean":"2024-04-24T00:38:45.911030+0000","last_became_active":"2024-04-24T00:38:34.632609+0000","last_became_peered":"2024-04-24T00:38:34.632609+0000","last_unstale":"2024-04-24T00:38:45.911030+0000","last_undegraded":"2024-04-24T00:38:45.911030+0000","last_fullsized":"2024-04-24T00:38:45.911030+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:51:37.284131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010187200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":560,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107558+0000","last_change":"2024-04-24T00:38:30.696852+0000","last_active":"2024-04-24T00:38:46.107558+0000","last_peered":"2024-04-24T00:38:46.107558+0000","last_clean":"2024-04-24T00:38:46.107558+0000","last_became_active":"2024-04-24T00:38:30.696223+0000","last_became_peered":"2024-04-24T00:38:30.696223+0000","last_unstale":"2024-04-24T00:38:46.107558+0000","last_undegraded":"2024-04-24T00:38:46.107558+0000","last_fullsized":"2024-04-24T00:38:46.107558+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:08:35.422980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6524,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063153+0000","last_change":"2024-04-24T00:38:18.803861+0000","last_active":"2024-04-24T00:38:46.063153+0000","last_peered":"2024-04-24T00:38:46.063153+0000","last_clean":"2024-04-24T00:38:46.063153+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:46.063153+0000","last_undegraded":"2024-04-24T00:38:46.063153+0000","last_fullsized":"2024-04-24T00:38:46.063153+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":116,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910548+0000","last_change":"2024-04-24T00:37:48.621131+0000","last_active":"2024-04-24T00:38:45.910548+0000","last_peered":"2024-04-24T00:38:45.910548+0000","last_clean":"2024-04-24T00:38:45.910548+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:45.910548+0000","last_undegraded":"2024-04-24T00:38:45.910548+0000","last_fullsized":"2024-04-24T00:38:45.910548+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4788,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440816+0000","last_change":"2024-04-24T00:38:36.639909+0000","last_active":"2024-04-24T00:38:47.440816+0000","last_peered":"2024-04-24T00:38:47.440816+0000","last_clean":"2024-04-24T00:38:47.440816+0000","last_became_active":"2024-04-24T00:38:36.637958+0000","last_became_peered":"2024-04-24T00:38:36.637958+0000","last_unstale":"2024-04-24T00:38:47.440816+0000","last_undegraded":"2024-04-24T00:38:47.440816+0000","last_fullsized":"2024-04-24T00:38:47.440816+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:23.662984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017453659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813411+0000","last_change":"2024-04-24T00:38:30.641161+0000","last_active":"2024-04-24T00:38:46.813411+0000","last_peered":"2024-04-24T00:38:46.813411+0000","last_clean":"2024-04-24T00:38:46.813411+0000","last_became_active":"2024-04-24T00:38:30.640288+0000","last_became_peered":"2024-04-24T00:38:30.640288+0000","last_unstale":"2024-04-24T00:38:46.813411+0000","last_undegraded":"2024-04-24T00:38:46.813411+0000","last_fullsized":"2024-04-24T00:38:46.813411+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:46:24.491722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4162,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910131+0000","last_change":"2024-04-24T00:38:18.780633+0000","last_active":"2024-04-24T00:38:45.910131+0000","last_peered":"2024-04-24T00:38:45.910131+0000","last_clean":"2024-04-24T00:38:45.910131+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:45.910131+0000","last_undegraded":"2024-04-24T00:38:45.910131+0000","last_fullsized":"2024-04-24T00:38:45.910131+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099788599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":124,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:53.431361+0000","last_change":"2024-04-24T00:38:53.431385+0000","last_active":"2024-04-24T00:38:53.431361+0000","last_peered":"2024-04-24T00:38:53.431361+0000","last_clean":"2024-04-24T00:38:53.431361+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:53.431361+0000","last_undegraded":"2024-04-24T00:38:53.431361+0000","last_fullsized":"2024-04-24T00:38:53.431361+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:53.431260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:53.431260+0000","last_clean_scrub_stamp":"2024-04-24T00:38:53.431260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:48:06.466836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6292,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911342+0000","last_change":"2024-04-24T00:38:18.767450+0000","last_active":"2024-04-24T00:38:45.911342+0000","last_peered":"2024-04-24T00:38:45.911342+0000","last_clean":"2024-04-24T00:38:45.911342+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:45.911342+0000","last_undegraded":"2024-04-24T00:38:45.911342+0000","last_fullsized":"2024-04-24T00:38:45.911342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:09:34.691573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"266'580","reported_seq":6804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910527+0000","last_change":"2024-04-24T00:38:32.649496+0000","last_active":"2024-04-24T00:38:45.910527+0000","last_peered":"2024-04-24T00:38:45.910527+0000","last_clean":"2024-04-24T00:38:45.910527+0000","last_became_active":"2024-04-24T00:38:32.645508+0000","last_became_peered":"2024-04-24T00:38:32.645508+0000","last_unstale":"2024-04-24T00:38:45.910527+0000","last_undegraded":"2024-04-24T00:38:45.910527+0000","last_fullsized":"2024-04-24T00:38:45.910527+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:10:59.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910983+0000","last_change":"2024-04-24T00:34:28.721983+0000","last_active":"2024-04-24T00:38:45.910983+0000","last_peered":"2024-04-24T00:38:45.910983+0000","last_clean":"2024-04-24T00:38:45.910983+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:45.910983+0000","last_undegraded":"2024-04-24T00:38:45.910983+0000","last_fullsized":"2024-04-24T00:38:45.910983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:06:06.153125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7976,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.909822+0000","last_change":"2024-04-24T00:38:18.780209+0000","last_active":"2024-04-24T00:38:45.909822+0000","last_peered":"2024-04-24T00:38:45.909822+0000","last_clean":"2024-04-24T00:38:45.909822+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:45.909822+0000","last_undegraded":"2024-04-24T00:38:45.909822+0000","last_fullsized":"2024-04-24T00:38:45.909822+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":551,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.370514+0000","last_change":"2024-04-24T00:33:44.958685+0000","last_active":"2024-04-24T00:38:46.370514+0000","last_peered":"2024-04-24T00:38:46.370514+0000","last_clean":"2024-04-24T00:38:46.370514+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:46.370514+0000","last_undegraded":"2024-04-24T00:38:46.370514+0000","last_fullsized":"2024-04-24T00:38:46.370514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":5036,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063127+0000","last_change":"2024-04-24T00:38:34.632199+0000","last_active":"2024-04-24T00:38:46.063127+0000","last_peered":"2024-04-24T00:38:46.063127+0000","last_clean":"2024-04-24T00:38:46.063127+0000","last_became_active":"2024-04-24T00:38:34.629456+0000","last_became_peered":"2024-04-24T00:38:34.629456+0000","last_unstale":"2024-04-24T00:38:46.063127+0000","last_undegraded":"2024-04-24T00:38:46.063127+0000","last_fullsized":"2024-04-24T00:38:46.063127+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813416+0000","last_change":"2024-04-24T00:38:30.634583+0000","last_active":"2024-04-24T00:38:46.813416+0000","last_peered":"2024-04-24T00:38:46.813416+0000","last_clean":"2024-04-24T00:38:46.813416+0000","last_became_active":"2024-04-24T00:38:30.633606+0000","last_became_peered":"2024-04-24T00:38:30.633606+0000","last_unstale":"2024-04-24T00:38:46.813416+0000","last_undegraded":"2024-04-24T00:38:46.813416+0000","last_fullsized":"2024-04-24T00:38:46.813416+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:24:57.510774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2875,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440503+0000","last_change":"2024-04-24T00:38:36.633952+0000","last_active":"2024-04-24T00:38:47.440503+0000","last_peered":"2024-04-24T00:38:47.440503+0000","last_clean":"2024-04-24T00:38:47.440503+0000","last_became_active":"2024-04-24T00:38:36.632239+0000","last_became_peered":"2024-04-24T00:38:36.632239+0000","last_unstale":"2024-04-24T00:38:47.440503+0000","last_undegraded":"2024-04-24T00:38:47.440503+0000","last_fullsized":"2024-04-24T00:38:47.440503+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:50.888554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012896030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910972+0000","last_change":"2024-04-24T00:34:28.718289+0000","last_active":"2024-04-24T00:38:45.910972+0000","last_peered":"2024-04-24T00:38:45.910972+0000","last_clean":"2024-04-24T00:38:45.910972+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:45.910972+0000","last_undegraded":"2024-04-24T00:38:45.910972+0000","last_fullsized":"2024-04-24T00:38:45.910972+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:43.935472+0000","last_clean_scrub_stamp":"2024-04-24T00:33:43.935472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:15:34.696893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":506,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813566+0000","last_change":"2024-04-24T00:38:36.642364+0000","last_active":"2024-04-24T00:38:46.813566+0000","last_peered":"2024-04-24T00:38:46.813566+0000","last_clean":"2024-04-24T00:38:46.813566+0000","last_became_active":"2024-04-24T00:38:36.641100+0000","last_became_peered":"2024-04-24T00:38:36.641100+0000","last_unstale":"2024-04-24T00:38:46.813566+0000","last_undegraded":"2024-04-24T00:38:46.813566+0000","last_fullsized":"2024-04-24T00:38:46.813566+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:39.898967+0000","last_clean_scrub_stamp":"2024-04-24T00:33:39.898967+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:00.586831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6985,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440455+0000","last_change":"2024-04-24T00:38:18.755623+0000","last_active":"2024-04-24T00:38:47.440455+0000","last_peered":"2024-04-24T00:38:47.440455+0000","last_clean":"2024-04-24T00:38:47.440455+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:47.440455+0000","last_undegraded":"2024-04-24T00:38:47.440455+0000","last_fullsized":"2024-04-24T00:38:47.440455+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:44.367052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"265'987","reported_seq":5319,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.910553+0000","last_change":"2024-04-24T00:38:33.656528+0000","last_active":"2024-04-24T00:38:45.910553+0000","last_peered":"2024-04-24T00:38:45.910553+0000","last_clean":"2024-04-24T00:38:45.910553+0000","last_became_active":"2024-04-24T00:38:33.650599+0000","last_became_peered":"2024-04-24T00:38:33.650599+0000","last_unstale":"2024-04-24T00:38:45.910553+0000","last_undegraded":"2024-04-24T00:38:45.910553+0000","last_fullsized":"2024-04-24T00:38:45.910553+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:19.834939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"263'1904","reported_seq":4804,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107528+0000","last_change":"2024-04-24T00:38:34.638496+0000","last_active":"2024-04-24T00:38:46.107528+0000","last_peered":"2024-04-24T00:38:46.107528+0000","last_clean":"2024-04-24T00:38:46.107528+0000","last_became_active":"2024-04-24T00:38:34.634199+0000","last_became_peered":"2024-04-24T00:38:34.634199+0000","last_unstale":"2024-04-24T00:38:46.107528+0000","last_undegraded":"2024-04-24T00:38:46.107528+0000","last_fullsized":"2024-04-24T00:38:46.107528+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:49:57.455108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001715414,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"263'461","reported_seq":4303,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:47.440544+0000","last_change":"2024-04-24T00:38:36.634741+0000","last_active":"2024-04-24T00:38:47.440544+0000","last_peered":"2024-04-24T00:38:47.440544+0000","last_clean":"2024-04-24T00:38:47.440544+0000","last_became_active":"2024-04-24T00:38:36.633468+0000","last_became_peered":"2024-04-24T00:38:36.633468+0000","last_unstale":"2024-04-24T00:38:47.440544+0000","last_undegraded":"2024-04-24T00:38:47.440544+0000","last_fullsized":"2024-04-24T00:38:47.440544+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:44:40.363226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002035177,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"266'831","reported_seq":10117,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:45.911887+0000","last_change":"2024-04-24T00:38:18.766586+0000","last_active":"2024-04-24T00:38:45.911887+0000","last_peered":"2024-04-24T00:38:45.911887+0000","last_clean":"2024-04-24T00:38:45.911887+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:45.911887+0000","last_undegraded":"2024-04-24T00:38:45.911887+0000","last_fullsized":"2024-04-24T00:38:45.911887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:34:15.463491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10600,"num_read_kb":68795,"num_write":1180,"num_write_kb":35219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"263'426","reported_seq":2956,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813960+0000","last_change":"2024-04-24T00:38:35.637609+0000","last_active":"2024-04-24T00:38:46.813960+0000","last_peered":"2024-04-24T00:38:46.813960+0000","last_clean":"2024-04-24T00:38:46.813960+0000","last_became_active":"2024-04-24T00:38:35.632277+0000","last_became_peered":"2024-04-24T00:38:35.632277+0000","last_unstale":"2024-04-24T00:38:46.813960+0000","last_undegraded":"2024-04-24T00:38:46.813960+0000","last_fullsized":"2024-04-24T00:38:46.813960+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:03.305609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"266'671","reported_seq":8093,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814115+0000","last_change":"2024-04-24T00:38:37.642821+0000","last_active":"2024-04-24T00:38:46.814115+0000","last_peered":"2024-04-24T00:38:46.814115+0000","last_clean":"2024-04-24T00:38:46.814115+0000","last_became_active":"2024-04-24T00:38:37.638998+0000","last_became_peered":"2024-04-24T00:38:37.638998+0000","last_unstale":"2024-04-24T00:38:46.814115+0000","last_undegraded":"2024-04-24T00:38:46.814115+0000","last_fullsized":"2024-04-24T00:38:46.814115+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:10:52.647284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001218907,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":952,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"263'874","reported_seq":5902,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.814247+0000","last_change":"2024-04-24T00:38:36.646741+0000","last_active":"2024-04-24T00:38:46.814247+0000","last_peered":"2024-04-24T00:38:46.814247+0000","last_clean":"2024-04-24T00:38:46.814247+0000","last_became_active":"2024-04-24T00:38:36.640462+0000","last_became_peered":"2024-04-24T00:38:36.640462+0000","last_unstale":"2024-04-24T00:38:46.814247+0000","last_undegraded":"2024-04-24T00:38:46.814247+0000","last_fullsized":"2024-04-24T00:38:46.814247+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:25.891296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"265'610","reported_seq":6427,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.107457+0000","last_change":"2024-04-24T00:38:18.844983+0000","last_active":"2024-04-24T00:38:46.107457+0000","last_peered":"2024-04-24T00:38:46.107457+0000","last_clean":"2024-04-24T00:38:46.107457+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:46.107457+0000","last_undegraded":"2024-04-24T00:38:46.107457+0000","last_fullsized":"2024-04-24T00:38:46.107457+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.135502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"265'598","reported_seq":8477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813747+0000","last_change":"2024-04-24T00:38:37.645213+0000","last_active":"2024-04-24T00:38:46.813747+0000","last_peered":"2024-04-24T00:38:46.813747+0000","last_clean":"2024-04-24T00:38:46.813747+0000","last_became_active":"2024-04-24T00:38:37.638797+0000","last_became_peered":"2024-04-24T00:38:37.638797+0000","last_unstale":"2024-04-24T00:38:46.813747+0000","last_undegraded":"2024-04-24T00:38:46.813747+0000","last_fullsized":"2024-04-24T00:38:46.813747+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:20.508035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"265'1515","reported_seq":6442,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.063101+0000","last_change":"2024-04-24T00:38:38.645480+0000","last_active":"2024-04-24T00:38:46.063101+0000","last_peered":"2024-04-24T00:38:46.063101+0000","last_clean":"2024-04-24T00:38:46.063101+0000","last_became_active":"2024-04-24T00:38:38.642317+0000","last_became_peered":"2024-04-24T00:38:38.642317+0000","last_unstale":"2024-04-24T00:38:46.063101+0000","last_undegraded":"2024-04-24T00:38:46.063101+0000","last_fullsized":"2024-04-24T00:38:46.063101+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"263'651","reported_seq":5104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:46.813680+0000","last_change":"2024-04-24T00:38:32.655883+0000","last_active":"2024-04-24T00:38:46.813680+0000","last_peered":"2024-04-24T00:38:46.813680+0000","last_clean":"2024-04-24T00:38:46.813680+0000","last_became_active":"2024-04-24T00:38:32.650425+0000","last_became_peered":"2024-04-24T00:38:32.650425+0000","last_unstale":"2024-04-24T00:38:46.813680+0000","last_undegraded":"2024-04-24T00:38:46.813680+0000","last_fullsized":"2024-04-24T00:38:46.813680+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:33:51.992273+0000","last_clean_scrub_stamp":"2024-04-24T00:33:51.992273+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:35:15.249883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013270599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185642,"num_read_kb":1545553,"num_write":31215,"num_write_kb":1018173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":358075070,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},"log_size":22825,"ondisk_log_size":22825,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":2951840,"data_compressed":34925,"data_compressed_allocated":184320,"data_compressed_original":2949120,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56860,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314980,"statfs":{"total":96636764160,"available":96578539520,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66548,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305292,"statfs":{"total":96636764160,"available":96568619008,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574915,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74872,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296968,"statfs":{"total":96636764160,"available":96560095232,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92924,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94278916,"statfs":{"total":96636764160,"available":96541609984,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74864,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296976,"statfs":{"total":96636764160,"available":96560103424,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60976,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310864,"statfs":{"total":96636764160,"available":96574324736,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60980,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310860,"statfs":{"total":96636764160,"available":96574320640,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893166,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39508,"kb_used_data":3468,"kb_used_omap":6,"kb_used_meta":36025,"kb_avail":94332332,"statfs":{"total":96636764160,"available":96596307968,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":36890339},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:38:56.481 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.481 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.481 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.481 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.481 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.481 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.481 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T00:33:43.935472+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T00:33:43.935472+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T00:33:43.935472+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.482 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.483 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.483 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.483 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T00:33:43.935472+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.483 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.483 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T00:33:43.935472+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.483 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.483 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T00:33:43.935472+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.483 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T00:33:43.935472+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T00:33:39.898967+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.484 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.485 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.485 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.485 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.485 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-24T00:33:51.992273+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=33, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=0, tm_min=38, tm_sec=51, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T00:38:56.485 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T00:39:16.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T00:39:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-24T00:39:16.803 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-24T00:39:16.823 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-04-24T00:39:15.737913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":185758,"num_read_kb":1545653,"num_write":31374,"num_write_kb":1020589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8595,"num_bytes_recovered":360436542,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22947,"ondisk_log_size":22947,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":527480,"kb_used_data":27512,"kb_used_omap":17,"kb_used_meta":499886,"kb_avail":754447240,"statfs":{"total":773094113280,"available":772553973760,"internally_reserved":0,"allocated":28172288,"data_stored":15156078,"data_compressed":13970,"data_compressed_allocated":73728,"data_compressed_original":1179648,"omap_allocated":17586,"internal_metadata":511884110},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003639"},"pg_stats":[{"pgid":"3.1f","version":"266'878","reported_seq":6008,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:54.535891+0000","last_change":"2024-04-24T00:38:54.535917+0000","last_active":"2024-04-24T00:38:54.535891+0000","last_peered":"2024-04-24T00:38:54.535891+0000","last_clean":"2024-04-24T00:38:54.535891+0000","last_became_active":"2024-04-24T00:37:47.617929+0000","last_became_peered":"2024-04-24T00:37:47.617929+0000","last_unstale":"2024-04-24T00:38:54.535891+0000","last_undegraded":"2024-04-24T00:38:54.535891+0000","last_fullsized":"2024-04-24T00:38:54.535891+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'878","last_scrub_stamp":"2024-04-24T00:38:54.535792+0000","last_deep_scrub":"266'878","last_deep_scrub_stamp":"2024-04-24T00:38:54.535792+0000","last_clean_scrub_stamp":"2024-04-24T00:38:54.535792+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:51:37.763435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010224170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6320,"num_read_kb":56004,"num_write":1346,"num_write_kb":38957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":15855946,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"264'1050","reported_seq":5694,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:39:00.417870+0000","last_change":"2024-04-24T00:39:00.417931+0000","last_active":"2024-04-24T00:39:00.417870+0000","last_peered":"2024-04-24T00:39:00.417870+0000","last_clean":"2024-04-24T00:39:00.417870+0000","last_became_active":"2024-04-24T00:38:36.639573+0000","last_became_peered":"2024-04-24T00:38:36.639573+0000","last_unstale":"2024-04-24T00:39:00.417870+0000","last_undegraded":"2024-04-24T00:39:00.417870+0000","last_fullsized":"2024-04-24T00:39:00.417870+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'1050","last_scrub_stamp":"2024-04-24T00:39:00.417662+0000","last_deep_scrub":"264'1050","last_deep_scrub_stamp":"2024-04-24T00:39:00.417662+0000","last_clean_scrub_stamp":"2024-04-24T00:39:00.417662+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:32:26.714241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00080331599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7957,"num_read_kb":35227,"num_write":1330,"num_write_kb":26877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"263'591","reported_seq":7556,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:39:00.349805+0000","last_change":"2024-04-24T00:39:00.349920+0000","last_active":"2024-04-24T00:39:00.349805+0000","last_peered":"2024-04-24T00:39:00.349805+0000","last_clean":"2024-04-24T00:39:00.349805+0000","last_became_active":"2024-04-24T00:38:33.649639+0000","last_became_peered":"2024-04-24T00:38:33.649639+0000","last_unstale":"2024-04-24T00:39:00.349805+0000","last_undegraded":"2024-04-24T00:39:00.349805+0000","last_fullsized":"2024-04-24T00:39:00.349805+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'591","last_scrub_stamp":"2024-04-24T00:39:00.349493+0000","last_deep_scrub":"263'591","last_deep_scrub_stamp":"2024-04-24T00:39:00.349493+0000","last_clean_scrub_stamp":"2024-04-24T00:39:00.349493+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:25:51.000783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0046012020000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7305,"num_read_kb":66853,"num_write":823,"num_write_kb":27402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":3658370,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"289'796","reported_seq":6347,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:39:01.368094+0000","last_change":"2024-04-24T00:39:01.368168+0000","last_active":"2024-04-24T00:39:01.368094+0000","last_peered":"2024-04-24T00:39:01.368094+0000","last_clean":"2024-04-24T00:39:01.368094+0000","last_became_active":"2024-04-24T00:38:35.628937+0000","last_became_peered":"2024-04-24T00:38:35.628937+0000","last_unstale":"2024-04-24T00:39:01.368094+0000","last_undegraded":"2024-04-24T00:39:01.368094+0000","last_fullsized":"2024-04-24T00:39:01.368094+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'796","last_scrub_stamp":"2024-04-24T00:39:01.367897+0000","last_deep_scrub":"289'796","last_deep_scrub_stamp":"2024-04-24T00:39:01.367897+0000","last_clean_scrub_stamp":"2024-04-24T00:39:01.367897+0000","objects_scrubbed":1,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:17:16.238851+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0045795660000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5266,"num_read_kb":61368,"num_write":1008,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":2421694,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"264'498","reported_seq":4565,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:57.264103+0000","last_change":"2024-04-24T00:38:57.264165+0000","last_active":"2024-04-24T00:38:57.264103+0000","last_peered":"2024-04-24T00:38:57.264103+0000","last_clean":"2024-04-24T00:38:57.264103+0000","last_became_active":"2024-04-24T00:34:28.730427+0000","last_became_peered":"2024-04-24T00:34:28.730427+0000","last_unstale":"2024-04-24T00:38:57.264103+0000","last_undegraded":"2024-04-24T00:38:57.264103+0000","last_fullsized":"2024-04-24T00:38:57.264103+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'498","last_scrub_stamp":"2024-04-24T00:38:57.263904+0000","last_deep_scrub":"264'498","last_deep_scrub_stamp":"2024-04-24T00:38:57.263904+0000","last_clean_scrub_stamp":"2024-04-24T00:38:57.263904+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:59.129430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013221800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4169,"num_read_kb":45609,"num_write":605,"num_write_kb":27389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1462269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"289'391","reported_seq":6320,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:58.390162+0000","last_change":"2024-04-24T00:38:58.390234+0000","last_active":"2024-04-24T00:38:58.390162+0000","last_peered":"2024-04-24T00:38:58.390162+0000","last_clean":"2024-04-24T00:38:58.390162+0000","last_became_active":"2024-04-24T00:38:35.631522+0000","last_became_peered":"2024-04-24T00:38:35.631522+0000","last_unstale":"2024-04-24T00:38:58.390162+0000","last_undegraded":"2024-04-24T00:38:58.390162+0000","last_fullsized":"2024-04-24T00:38:58.390162+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'391","last_scrub_stamp":"2024-04-24T00:38:58.389978+0000","last_deep_scrub":"289'391","last_deep_scrub_stamp":"2024-04-24T00:38:58.389978+0000","last_clean_scrub_stamp":"2024-04-24T00:38:58.389978+0000","objects_scrubbed":1,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:10:14.917647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001287217,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4509,"num_read_kb":37330,"num_write":498,"num_write_kb":25759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":3953416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"265'712","reported_seq":5918,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:56.443249+0000","last_change":"2024-04-24T00:38:56.443399+0000","last_active":"2024-04-24T00:38:56.443249+0000","last_peered":"2024-04-24T00:38:56.443249+0000","last_clean":"2024-04-24T00:38:56.443249+0000","last_became_active":"2024-04-24T00:37:52.613606+0000","last_became_peered":"2024-04-24T00:37:52.613606+0000","last_unstale":"2024-04-24T00:38:56.443249+0000","last_undegraded":"2024-04-24T00:38:56.443249+0000","last_fullsized":"2024-04-24T00:38:56.443249+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'712","last_scrub_stamp":"2024-04-24T00:38:56.443001+0000","last_deep_scrub":"265'712","last_deep_scrub_stamp":"2024-04-24T00:38:56.443001+0000","last_clean_scrub_stamp":"2024-04-24T00:38:56.443001+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:12:59.671660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059221100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5963,"num_read_kb":40407,"num_write":1004,"num_write_kb":29820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":1722153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"263'551","reported_seq":4888,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:56.468993+0000","last_change":"2024-04-24T00:38:56.469024+0000","last_active":"2024-04-24T00:38:56.468993+0000","last_peered":"2024-04-24T00:38:56.468993+0000","last_clean":"2024-04-24T00:38:56.468993+0000","last_became_active":"2024-04-24T00:38:36.631395+0000","last_became_peered":"2024-04-24T00:38:36.631395+0000","last_unstale":"2024-04-24T00:38:56.468993+0000","last_undegraded":"2024-04-24T00:38:56.468993+0000","last_fullsized":"2024-04-24T00:38:56.468993+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'551","last_scrub_stamp":"2024-04-24T00:38:56.468874+0000","last_deep_scrub":"263'551","last_deep_scrub_stamp":"2024-04-24T00:38:56.468874+0000","last_clean_scrub_stamp":"2024-04-24T00:38:56.468874+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:55:57.052192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001058769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5363,"num_read_kb":36889,"num_write":597,"num_write_kb":26573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":20066347,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"263'339","reported_seq":4822,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:53.508792+0000","last_change":"2024-04-24T00:38:53.508850+0000","last_active":"2024-04-24T00:38:53.508792+0000","last_peered":"2024-04-24T00:38:53.508792+0000","last_clean":"2024-04-24T00:38:53.508792+0000","last_became_active":"2024-04-24T00:37:46.614071+0000","last_became_peered":"2024-04-24T00:37:46.614071+0000","last_unstale":"2024-04-24T00:38:53.508792+0000","last_undegraded":"2024-04-24T00:38:53.508792+0000","last_fullsized":"2024-04-24T00:38:53.508792+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'339","last_scrub_stamp":"2024-04-24T00:38:53.508594+0000","last_deep_scrub":"263'339","last_deep_scrub_stamp":"2024-04-24T00:38:53.508594+0000","last_clean_scrub_stamp":"2024-04-24T00:38:53.508594+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:05:28.524601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090935799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3230,"num_read_kb":40177,"num_write":428,"num_write_kb":23417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":15603987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"266'751","reported_seq":6825,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:54.453173+0000","last_change":"2024-04-24T00:38:54.453219+0000","last_active":"2024-04-24T00:38:54.453173+0000","last_peered":"2024-04-24T00:38:54.453173+0000","last_clean":"2024-04-24T00:38:54.453173+0000","last_became_active":"2024-04-24T00:35:35.560309+0000","last_became_peered":"2024-04-24T00:35:35.560309+0000","last_unstale":"2024-04-24T00:38:54.453173+0000","last_undegraded":"2024-04-24T00:38:54.453173+0000","last_fullsized":"2024-04-24T00:38:54.453173+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'751","last_scrub_stamp":"2024-04-24T00:38:54.453066+0000","last_deep_scrub":"266'751","last_deep_scrub_stamp":"2024-04-24T00:38:54.453066+0000","last_clean_scrub_stamp":"2024-04-24T00:38:54.453066+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:11:12.014768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012715420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7650,"num_read_kb":50977,"num_write":1087,"num_write_kb":38925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":6440279,"num_keys_recovered":51,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"264'632","reported_seq":7284,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:58.938585+0000","last_change":"2024-04-24T00:38:58.938643+0000","last_active":"2024-04-24T00:38:58.938585+0000","last_peered":"2024-04-24T00:38:58.938585+0000","last_clean":"2024-04-24T00:38:58.938585+0000","last_became_active":"2024-04-24T00:38:32.643949+0000","last_became_peered":"2024-04-24T00:38:32.643949+0000","last_unstale":"2024-04-24T00:38:58.938585+0000","last_undegraded":"2024-04-24T00:38:58.938585+0000","last_fullsized":"2024-04-24T00:38:58.938585+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'632","last_scrub_stamp":"2024-04-24T00:38:58.938380+0000","last_deep_scrub":"264'632","last_deep_scrub_stamp":"2024-04-24T00:38:58.938380+0000","last_clean_scrub_stamp":"2024-04-24T00:38:58.938380+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:37:43.175362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013149710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":55669,"num_write":895,"num_write_kb":31185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":3105895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"264'496","reported_seq":5392,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:57.397113+0000","last_change":"2024-04-24T00:38:57.397173+0000","last_active":"2024-04-24T00:38:57.397113+0000","last_peered":"2024-04-24T00:38:57.397113+0000","last_clean":"2024-04-24T00:38:57.397113+0000","last_became_active":"2024-04-24T00:38:34.632609+0000","last_became_peered":"2024-04-24T00:38:34.632609+0000","last_unstale":"2024-04-24T00:38:57.397113+0000","last_undegraded":"2024-04-24T00:38:57.397113+0000","last_fullsized":"2024-04-24T00:38:57.397113+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'496","last_scrub_stamp":"2024-04-24T00:38:57.396911+0000","last_deep_scrub":"264'496","last_deep_scrub_stamp":"2024-04-24T00:38:57.396911+0000","last_clean_scrub_stamp":"2024-04-24T00:38:57.396911+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:21.580325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010187200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4143,"num_read_kb":41581,"num_write":581,"num_write_kb":30982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":3597588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":569,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:57.232156+0000","last_change":"2024-04-24T00:38:57.232212+0000","last_active":"2024-04-24T00:38:57.232156+0000","last_peered":"2024-04-24T00:38:57.232156+0000","last_clean":"2024-04-24T00:38:57.232156+0000","last_became_active":"2024-04-24T00:38:30.696223+0000","last_became_peered":"2024-04-24T00:38:30.696223+0000","last_unstale":"2024-04-24T00:38:57.232156+0000","last_undegraded":"2024-04-24T00:38:57.232156+0000","last_fullsized":"2024-04-24T00:38:57.232156+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:57.231950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:57.231950+0000","last_clean_scrub_stamp":"2024-04-24T00:38:57.231950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:19.607968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"266'580","reported_seq":6532,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:55.582602+0000","last_change":"2024-04-24T00:38:55.582666+0000","last_active":"2024-04-24T00:38:55.582602+0000","last_peered":"2024-04-24T00:38:55.582602+0000","last_clean":"2024-04-24T00:38:55.582602+0000","last_became_active":"2024-04-24T00:37:51.617280+0000","last_became_peered":"2024-04-24T00:37:51.617280+0000","last_unstale":"2024-04-24T00:38:55.582602+0000","last_undegraded":"2024-04-24T00:38:55.582602+0000","last_fullsized":"2024-04-24T00:38:55.582602+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'580","last_scrub_stamp":"2024-04-24T00:38:55.582400+0000","last_deep_scrub":"266'580","last_deep_scrub_stamp":"2024-04-24T00:38:55.582400+0000","last_clean_scrub_stamp":"2024-04-24T00:38:55.582400+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:37.286213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00076055700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5118,"num_read_kb":58059,"num_write":896,"num_write_kb":42139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":48309510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":124,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:54.473791+0000","last_change":"2024-04-24T00:38:54.473812+0000","last_active":"2024-04-24T00:38:54.473791+0000","last_peered":"2024-04-24T00:38:54.473791+0000","last_clean":"2024-04-24T00:38:54.473791+0000","last_became_active":"2024-04-24T00:37:48.620867+0000","last_became_peered":"2024-04-24T00:37:48.620867+0000","last_unstale":"2024-04-24T00:38:54.473791+0000","last_undegraded":"2024-04-24T00:38:54.473791+0000","last_fullsized":"2024-04-24T00:38:54.473791+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:54.473708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:54.473708+0000","last_clean_scrub_stamp":"2024-04-24T00:38:54.473708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:00.009652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"262'620","reported_seq":4797,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:59.457382+0000","last_change":"2024-04-24T00:38:59.457449+0000","last_active":"2024-04-24T00:38:59.457382+0000","last_peered":"2024-04-24T00:38:59.457382+0000","last_clean":"2024-04-24T00:38:59.457382+0000","last_became_active":"2024-04-24T00:38:36.637958+0000","last_became_peered":"2024-04-24T00:38:36.637958+0000","last_unstale":"2024-04-24T00:38:59.457382+0000","last_undegraded":"2024-04-24T00:38:59.457382+0000","last_fullsized":"2024-04-24T00:38:59.457382+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'620","last_scrub_stamp":"2024-04-24T00:38:59.457168+0000","last_deep_scrub":"262'620","last_deep_scrub_stamp":"2024-04-24T00:38:59.457168+0000","last_clean_scrub_stamp":"2024-04-24T00:38:59.457168+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:05:16.827053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0017453659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4511,"num_read_kb":41807,"num_write":840,"num_write_kb":26247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":28314227,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:57.910004+0000","last_change":"2024-04-24T00:38:57.910063+0000","last_active":"2024-04-24T00:38:57.910004+0000","last_peered":"2024-04-24T00:38:57.910004+0000","last_clean":"2024-04-24T00:38:57.910004+0000","last_became_active":"2024-04-24T00:38:30.640288+0000","last_became_peered":"2024-04-24T00:38:30.640288+0000","last_unstale":"2024-04-24T00:38:57.910004+0000","last_undegraded":"2024-04-24T00:38:57.910004+0000","last_fullsized":"2024-04-24T00:38:57.910004+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:57.909805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:57.909805+0000","last_clean_scrub_stamp":"2024-04-24T00:38:57.909805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:07:20.183583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"265'529","reported_seq":4170,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:55.445377+0000","last_change":"2024-04-24T00:38:55.445400+0000","last_active":"2024-04-24T00:38:55.445377+0000","last_peered":"2024-04-24T00:38:55.445377+0000","last_clean":"2024-04-24T00:38:55.445377+0000","last_became_active":"2024-04-24T00:37:52.609923+0000","last_became_peered":"2024-04-24T00:37:52.609923+0000","last_unstale":"2024-04-24T00:38:55.445377+0000","last_undegraded":"2024-04-24T00:38:55.445377+0000","last_fullsized":"2024-04-24T00:38:55.445377+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'529","last_scrub_stamp":"2024-04-24T00:38:55.445302+0000","last_deep_scrub":"265'529","last_deep_scrub_stamp":"2024-04-24T00:38:55.445302+0000","last_clean_scrub_stamp":"2024-04-24T00:38:55.445302+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:24:35.700939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00099788599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3467,"num_read_kb":34433,"num_write":732,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3158730,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":124,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:53.431361+0000","last_change":"2024-04-24T00:38:53.431385+0000","last_active":"2024-04-24T00:38:53.431361+0000","last_peered":"2024-04-24T00:38:53.431361+0000","last_clean":"2024-04-24T00:38:53.431361+0000","last_became_active":"2024-04-24T00:37:48.620276+0000","last_became_peered":"2024-04-24T00:37:48.620276+0000","last_unstale":"2024-04-24T00:38:53.431361+0000","last_undegraded":"2024-04-24T00:38:53.431361+0000","last_fullsized":"2024-04-24T00:38:53.431361+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:53.431260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:53.431260+0000","last_clean_scrub_stamp":"2024-04-24T00:38:53.431260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:48:06.466836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"265'667","reported_seq":6301,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:58.291847+0000","last_change":"2024-04-24T00:38:58.291910+0000","last_active":"2024-04-24T00:38:58.291847+0000","last_peered":"2024-04-24T00:38:58.291847+0000","last_clean":"2024-04-24T00:38:58.291847+0000","last_became_active":"2024-04-24T00:34:31.750679+0000","last_became_peered":"2024-04-24T00:34:31.750679+0000","last_unstale":"2024-04-24T00:38:58.291847+0000","last_undegraded":"2024-04-24T00:38:58.291847+0000","last_fullsized":"2024-04-24T00:38:58.291847+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'667","last_scrub_stamp":"2024-04-24T00:38:58.291625+0000","last_deep_scrub":"265'667","last_deep_scrub_stamp":"2024-04-24T00:38:58.291625+0000","last_clean_scrub_stamp":"2024-04-24T00:38:58.291625+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:12:21.796675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0038953339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6178,"num_read_kb":55659,"num_write":842,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":128134,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"289'581","reported_seq":6814,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:55.439681+0000","last_change":"2024-04-24T00:38:55.439754+0000","last_active":"2024-04-24T00:38:55.439681+0000","last_peered":"2024-04-24T00:38:55.439681+0000","last_clean":"2024-04-24T00:38:55.439681+0000","last_became_active":"2024-04-24T00:38:32.645508+0000","last_became_peered":"2024-04-24T00:38:32.645508+0000","last_unstale":"2024-04-24T00:38:55.439681+0000","last_undegraded":"2024-04-24T00:38:55.439681+0000","last_fullsized":"2024-04-24T00:38:55.439681+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'581","last_scrub_stamp":"2024-04-24T00:38:55.439494+0000","last_deep_scrub":"289'581","last_deep_scrub_stamp":"2024-04-24T00:38:55.439494+0000","last_clean_scrub_stamp":"2024-04-24T00:38:55.439494+0000","objects_scrubbed":1,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:56:30.141713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052705600000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5091,"num_read_kb":51505,"num_write":742,"num_write_kb":30886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":30016470,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":490,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:56.300417+0000","last_change":"2024-04-24T00:38:56.300472+0000","last_active":"2024-04-24T00:38:56.300417+0000","last_peered":"2024-04-24T00:38:56.300417+0000","last_clean":"2024-04-24T00:38:56.300417+0000","last_became_active":"2024-04-24T00:34:28.721185+0000","last_became_peered":"2024-04-24T00:34:28.721185+0000","last_unstale":"2024-04-24T00:38:56.300417+0000","last_undegraded":"2024-04-24T00:38:56.300417+0000","last_fullsized":"2024-04-24T00:38:56.300417+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:56.300215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:56.300215+0000","last_clean_scrub_stamp":"2024-04-24T00:38:56.300215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:30.624445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"264'656","reported_seq":7984,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:57.400490+0000","last_change":"2024-04-24T00:38:57.400572+0000","last_active":"2024-04-24T00:38:57.400490+0000","last_peered":"2024-04-24T00:38:57.400490+0000","last_clean":"2024-04-24T00:38:57.400490+0000","last_became_active":"2024-04-24T00:37:53.611229+0000","last_became_peered":"2024-04-24T00:37:53.611229+0000","last_unstale":"2024-04-24T00:38:57.400490+0000","last_undegraded":"2024-04-24T00:38:57.400490+0000","last_fullsized":"2024-04-24T00:38:57.400490+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'656","last_scrub_stamp":"2024-04-24T00:38:57.400273+0000","last_deep_scrub":"264'656","last_deep_scrub_stamp":"2024-04-24T00:38:57.400273+0000","last_clean_scrub_stamp":"2024-04-24T00:38:57.400273+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:24:51.971200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7628,"num_read_kb":66033,"num_write":721,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":4762724,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":559,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:53.814781+0000","last_change":"2024-04-24T00:38:53.814866+0000","last_active":"2024-04-24T00:38:53.814781+0000","last_peered":"2024-04-24T00:38:53.814781+0000","last_clean":"2024-04-24T00:38:53.814781+0000","last_became_active":"2024-04-24T00:33:44.957884+0000","last_became_peered":"2024-04-24T00:33:44.957884+0000","last_unstale":"2024-04-24T00:38:53.814781+0000","last_undegraded":"2024-04-24T00:38:53.814781+0000","last_fullsized":"2024-04-24T00:38:53.814781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:53.814634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:53.814634+0000","last_clean_scrub_stamp":"2024-04-24T00:38:53.814634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:28:04.730509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"264'444","reported_seq":5044,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:56.573660+0000","last_change":"2024-04-24T00:38:56.573801+0000","last_active":"2024-04-24T00:38:56.573660+0000","last_peered":"2024-04-24T00:38:56.573660+0000","last_clean":"2024-04-24T00:38:56.573660+0000","last_became_active":"2024-04-24T00:38:34.629456+0000","last_became_peered":"2024-04-24T00:38:34.629456+0000","last_unstale":"2024-04-24T00:38:56.573660+0000","last_undegraded":"2024-04-24T00:38:56.573660+0000","last_fullsized":"2024-04-24T00:38:56.573660+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'444","last_scrub_stamp":"2024-04-24T00:38:56.573387+0000","last_deep_scrub":"264'444","last_deep_scrub_stamp":"2024-04-24T00:38:56.573387+0000","last_clean_scrub_stamp":"2024-04-24T00:38:56.573387+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:34:56.780189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00061895799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2121,"num_read_kb":33776,"num_write":556,"num_write_kb":25140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":32335223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:56.923599+0000","last_change":"2024-04-24T00:38:56.923652+0000","last_active":"2024-04-24T00:38:56.923599+0000","last_peered":"2024-04-24T00:38:56.923599+0000","last_clean":"2024-04-24T00:38:56.923599+0000","last_became_active":"2024-04-24T00:38:30.633606+0000","last_became_peered":"2024-04-24T00:38:30.633606+0000","last_unstale":"2024-04-24T00:38:56.923599+0000","last_undegraded":"2024-04-24T00:38:56.923599+0000","last_fullsized":"2024-04-24T00:38:56.923599+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:56.923402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:56.923402+0000","last_clean_scrub_stamp":"2024-04-24T00:38:56.923402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:29:07.499354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"265'609","reported_seq":2884,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:57.497425+0000","last_change":"2024-04-24T00:38:57.497496+0000","last_active":"2024-04-24T00:38:57.497425+0000","last_peered":"2024-04-24T00:38:57.497425+0000","last_clean":"2024-04-24T00:38:57.497425+0000","last_became_active":"2024-04-24T00:38:36.632239+0000","last_became_peered":"2024-04-24T00:38:36.632239+0000","last_unstale":"2024-04-24T00:38:57.497425+0000","last_undegraded":"2024-04-24T00:38:57.497425+0000","last_fullsized":"2024-04-24T00:38:57.497425+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'609","last_scrub_stamp":"2024-04-24T00:38:57.497207+0000","last_deep_scrub":"265'609","last_deep_scrub_stamp":"2024-04-24T00:38:57.497207+0000","last_clean_scrub_stamp":"2024-04-24T00:38:57.497207+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:59:54.165923+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0012896030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2765,"num_read_kb":24323,"num_write":779,"num_write_kb":27949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":345340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":490,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:55.347496+0000","last_change":"2024-04-24T00:38:55.347552+0000","last_active":"2024-04-24T00:38:55.347496+0000","last_peered":"2024-04-24T00:38:55.347496+0000","last_clean":"2024-04-24T00:38:55.347496+0000","last_became_active":"2024-04-24T00:34:28.717909+0000","last_became_peered":"2024-04-24T00:34:28.717909+0000","last_unstale":"2024-04-24T00:38:55.347496+0000","last_undegraded":"2024-04-24T00:38:55.347496+0000","last_fullsized":"2024-04-24T00:38:55.347496+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T00:38:55.347293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T00:38:55.347293+0000","last_clean_scrub_stamp":"2024-04-24T00:38:55.347293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:17:11.261996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"289'119","reported_seq":517,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:39:01.880827+0000","last_change":"2024-04-24T00:39:01.880960+0000","last_active":"2024-04-24T00:39:01.880827+0000","last_peered":"2024-04-24T00:39:01.880827+0000","last_clean":"2024-04-24T00:39:01.880827+0000","last_became_active":"2024-04-24T00:38:36.641100+0000","last_became_peered":"2024-04-24T00:38:36.641100+0000","last_unstale":"2024-04-24T00:39:01.880827+0000","last_undegraded":"2024-04-24T00:39:01.880827+0000","last_fullsized":"2024-04-24T00:39:01.880827+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'119","last_scrub_stamp":"2024-04-24T00:39:01.880560+0000","last_deep_scrub":"289'119","last_deep_scrub_stamp":"2024-04-24T00:39:01.880560+0000","last_clean_scrub_stamp":"2024-04-24T00:39:01.880560+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:43:42.877043+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"263'929","reported_seq":6994,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:55.498063+0000","last_change":"2024-04-24T00:38:55.498085+0000","last_active":"2024-04-24T00:38:55.498063+0000","last_peered":"2024-04-24T00:38:55.498063+0000","last_clean":"2024-04-24T00:38:55.498063+0000","last_became_active":"2024-04-24T00:37:53.610728+0000","last_became_peered":"2024-04-24T00:37:53.610728+0000","last_unstale":"2024-04-24T00:38:55.498063+0000","last_undegraded":"2024-04-24T00:38:55.498063+0000","last_fullsized":"2024-04-24T00:38:55.498063+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'929","last_scrub_stamp":"2024-04-24T00:38:55.497980+0000","last_deep_scrub":"263'929","last_deep_scrub_stamp":"2024-04-24T00:38:55.497980+0000","last_clean_scrub_stamp":"2024-04-24T00:38:55.497980+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:47:17.911784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073615599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7450,"num_read_kb":68974,"num_write":1246,"num_write_kb":32544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":8127789,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"265'987","reported_seq":5328,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:56.442900+0000","last_change":"2024-04-24T00:38:56.443059+0000","last_active":"2024-04-24T00:38:56.442900+0000","last_peered":"2024-04-24T00:38:56.442900+0000","last_clean":"2024-04-24T00:38:56.442900+0000","last_became_active":"2024-04-24T00:38:33.650599+0000","last_became_peered":"2024-04-24T00:38:33.650599+0000","last_unstale":"2024-04-24T00:38:56.442900+0000","last_undegraded":"2024-04-24T00:38:56.442900+0000","last_fullsized":"2024-04-24T00:38:56.442900+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'987","last_scrub_stamp":"2024-04-24T00:38:56.442683+0000","last_deep_scrub":"265'987","last_deep_scrub_stamp":"2024-04-24T00:38:56.442683+0000","last_clean_scrub_stamp":"2024-04-24T00:38:56.442683+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:24:05.673293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002179149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7299,"num_read_kb":32746,"num_write":1239,"num_write_kb":21159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2131240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"263'1904","reported_seq":4813,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:58.260774+0000","last_change":"2024-04-24T00:38:58.260837+0000","last_active":"2024-04-24T00:38:58.260774+0000","last_peered":"2024-04-24T00:38:58.260774+0000","last_clean":"2024-04-24T00:38:58.260774+0000","last_became_active":"2024-04-24T00:38:34.634199+0000","last_became_peered":"2024-04-24T00:38:34.634199+0000","last_unstale":"2024-04-24T00:38:58.260774+0000","last_undegraded":"2024-04-24T00:38:58.260774+0000","last_fullsized":"2024-04-24T00:38:58.260774+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'1904","last_scrub_stamp":"2024-04-24T00:38:58.260570+0000","last_deep_scrub":"263'1904","last_deep_scrub_stamp":"2024-04-24T00:38:58.260570+0000","last_clean_scrub_stamp":"2024-04-24T00:38:58.260570+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:57:16.701461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001715414,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9571,"num_read_kb":44939,"num_write":3261,"num_write_kb":33625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":1801107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"263'461","reported_seq":4312,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:58.446931+0000","last_change":"2024-04-24T00:38:58.446992+0000","last_active":"2024-04-24T00:38:58.446931+0000","last_peered":"2024-04-24T00:38:58.446931+0000","last_clean":"2024-04-24T00:38:58.446931+0000","last_became_active":"2024-04-24T00:38:36.633468+0000","last_became_peered":"2024-04-24T00:38:36.633468+0000","last_unstale":"2024-04-24T00:38:58.446931+0000","last_undegraded":"2024-04-24T00:38:58.446931+0000","last_fullsized":"2024-04-24T00:38:58.446931+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'461","last_scrub_stamp":"2024-04-24T00:38:58.446671+0000","last_deep_scrub":"263'461","last_deep_scrub_stamp":"2024-04-24T00:38:58.446671+0000","last_clean_scrub_stamp":"2024-04-24T00:38:58.446671+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:00:29.414763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002035177,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3989,"num_read_kb":34789,"num_write":475,"num_write_kb":21906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":15230281,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"266'831","reported_seq":10127,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:59.313410+0000","last_change":"2024-04-24T00:38:59.313473+0000","last_active":"2024-04-24T00:38:59.313410+0000","last_peered":"2024-04-24T00:38:59.313410+0000","last_clean":"2024-04-24T00:38:59.313410+0000","last_became_active":"2024-04-24T00:34:31.752180+0000","last_became_peered":"2024-04-24T00:34:31.752180+0000","last_unstale":"2024-04-24T00:38:59.313410+0000","last_undegraded":"2024-04-24T00:38:59.313410+0000","last_fullsized":"2024-04-24T00:38:59.313410+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'831","last_scrub_stamp":"2024-04-24T00:38:59.313192+0000","last_deep_scrub":"266'831","last_deep_scrub_stamp":"2024-04-24T00:38:59.313192+0000","last_clean_scrub_stamp":"2024-04-24T00:38:59.313192+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:36:56.511279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031150959999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10600,"num_read_kb":68795,"num_write":1180,"num_write_kb":35219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":405111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"263'426","reported_seq":2965,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:39:00.886140+0000","last_change":"2024-04-24T00:39:00.886206+0000","last_active":"2024-04-24T00:39:00.886140+0000","last_peered":"2024-04-24T00:39:00.886140+0000","last_clean":"2024-04-24T00:39:00.886140+0000","last_became_active":"2024-04-24T00:38:35.632277+0000","last_became_peered":"2024-04-24T00:38:35.632277+0000","last_unstale":"2024-04-24T00:39:00.886140+0000","last_undegraded":"2024-04-24T00:39:00.886140+0000","last_fullsized":"2024-04-24T00:39:00.886140+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'426","last_scrub_stamp":"2024-04-24T00:39:00.885935+0000","last_deep_scrub":"263'426","last_deep_scrub_stamp":"2024-04-24T00:39:00.885935+0000","last_clean_scrub_stamp":"2024-04-24T00:39:00.885935+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:58:10.749900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0033277850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1997,"num_read_kb":38257,"num_write":553,"num_write_kb":30693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1749171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"266'671","reported_seq":8102,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:39:03.843239+0000","last_change":"2024-04-24T00:39:03.843301+0000","last_active":"2024-04-24T00:39:03.843239+0000","last_peered":"2024-04-24T00:39:03.843239+0000","last_clean":"2024-04-24T00:39:03.843239+0000","last_became_active":"2024-04-24T00:38:37.638998+0000","last_became_peered":"2024-04-24T00:38:37.638998+0000","last_unstale":"2024-04-24T00:39:03.843239+0000","last_undegraded":"2024-04-24T00:39:03.843239+0000","last_fullsized":"2024-04-24T00:39:03.843239+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'671","last_scrub_stamp":"2024-04-24T00:39:03.843007+0000","last_deep_scrub":"266'671","last_deep_scrub_stamp":"2024-04-24T00:39:03.843007+0000","last_clean_scrub_stamp":"2024-04-24T00:39:03.843007+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:34:58.840489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001218907,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7847,"num_read_kb":69081,"num_write":952,"num_write_kb":38895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":5305865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"263'874","reported_seq":5911,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:39:02.846653+0000","last_change":"2024-04-24T00:39:02.846711+0000","last_active":"2024-04-24T00:39:02.846653+0000","last_peered":"2024-04-24T00:39:02.846653+0000","last_clean":"2024-04-24T00:39:02.846653+0000","last_became_active":"2024-04-24T00:38:36.640462+0000","last_became_peered":"2024-04-24T00:38:36.640462+0000","last_unstale":"2024-04-24T00:39:02.846653+0000","last_undegraded":"2024-04-24T00:39:02.846653+0000","last_fullsized":"2024-04-24T00:39:02.846653+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'874","last_scrub_stamp":"2024-04-24T00:39:02.846468+0000","last_deep_scrub":"263'874","last_deep_scrub_stamp":"2024-04-24T00:39:02.846468+0000","last_clean_scrub_stamp":"2024-04-24T00:39:02.846468+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:19:02.022155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011647039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5546,"num_read_kb":46075,"num_write":1371,"num_write_kb":43860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":28638438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"265'610","reported_seq":6436,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:56.218761+0000","last_change":"2024-04-24T00:38:56.218828+0000","last_active":"2024-04-24T00:38:56.218761+0000","last_peered":"2024-04-24T00:38:56.218761+0000","last_clean":"2024-04-24T00:38:56.218761+0000","last_became_active":"2024-04-24T00:37:46.612939+0000","last_became_peered":"2024-04-24T00:37:46.612939+0000","last_unstale":"2024-04-24T00:38:56.218761+0000","last_undegraded":"2024-04-24T00:38:56.218761+0000","last_fullsized":"2024-04-24T00:38:56.218761+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'610","last_scrub_stamp":"2024-04-24T00:38:56.218557+0000","last_deep_scrub":"265'610","last_deep_scrub_stamp":"2024-04-24T00:38:56.218557+0000","last_clean_scrub_stamp":"2024-04-24T00:38:56.218557+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:52:53.126524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001057452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6185,"num_read_kb":30624,"num_write":709,"num_write_kb":20586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":14895414,"num_keys_recovered":66,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"265'598","reported_seq":8486,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:39:04.837865+0000","last_change":"2024-04-24T00:39:04.837926+0000","last_active":"2024-04-24T00:39:04.837865+0000","last_peered":"2024-04-24T00:39:04.837865+0000","last_clean":"2024-04-24T00:39:04.837865+0000","last_became_active":"2024-04-24T00:38:37.638797+0000","last_became_peered":"2024-04-24T00:38:37.638797+0000","last_unstale":"2024-04-24T00:39:04.837865+0000","last_undegraded":"2024-04-24T00:39:04.837865+0000","last_fullsized":"2024-04-24T00:39:04.837865+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'598","last_scrub_stamp":"2024-04-24T00:39:04.837654+0000","last_deep_scrub":"265'598","last_deep_scrub_stamp":"2024-04-24T00:39:04.837654+0000","last_clean_scrub_stamp":"2024-04-24T00:39:04.837654+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:13.360576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018953869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7413,"num_read_kb":84425,"num_write":695,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":2385220,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"265'1515","reported_seq":6450,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:57.621691+0000","last_change":"2024-04-24T00:38:57.621753+0000","last_active":"2024-04-24T00:38:57.621691+0000","last_peered":"2024-04-24T00:38:57.621691+0000","last_clean":"2024-04-24T00:38:57.621691+0000","last_became_active":"2024-04-24T00:38:38.642317+0000","last_became_peered":"2024-04-24T00:38:38.642317+0000","last_unstale":"2024-04-24T00:38:57.621691+0000","last_undegraded":"2024-04-24T00:38:57.621691+0000","last_fullsized":"2024-04-24T00:38:57.621691+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'1515","last_scrub_stamp":"2024-04-24T00:38:57.621436+0000","last_deep_scrub":"265'1515","last_deep_scrub_stamp":"2024-04-24T00:38:57.621436+0000","last_clean_scrub_stamp":"2024-04-24T00:38:57.621436+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:20:12.231762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00048223199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7638,"num_read_kb":53991,"num_write":2299,"num_write_kb":43783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":50522141,"num_keys_recovered":87,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"263'651","reported_seq":5113,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-24T00:38:59.916948+0000","last_change":"2024-04-24T00:38:59.917008+0000","last_active":"2024-04-24T00:38:59.916948+0000","last_peered":"2024-04-24T00:38:59.916948+0000","last_clean":"2024-04-24T00:38:59.916948+0000","last_became_active":"2024-04-24T00:38:32.650425+0000","last_became_peered":"2024-04-24T00:38:32.650425+0000","last_unstale":"2024-04-24T00:38:59.916948+0000","last_undegraded":"2024-04-24T00:38:59.916948+0000","last_fullsized":"2024-04-24T00:38:59.916948+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'651","last_scrub_stamp":"2024-04-24T00:38:59.916730+0000","last_deep_scrub":"263'651","last_deep_scrub_stamp":"2024-04-24T00:38:59.916730+0000","last_clean_scrub_stamp":"2024-04-24T00:38:59.916730+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:47.375568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013270599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":39171,"num_write":925,"num_write_kb":31087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1624991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185642,"num_read_kb":1545553,"num_write":31215,"num_write_kb":1018173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8587,"num_bytes_recovered":358075070,"num_keys_recovered":554,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},"log_size":22828,"ondisk_log_size":22828,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":2951840,"data_compressed":34925,"data_compressed_allocated":184320,"data_compressed_original":2949120,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574921,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56852,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314988,"statfs":{"total":96636764160,"available":96578547712,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574920,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66540,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305300,"statfs":{"total":96636764160,"available":96568627200,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574919,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74860,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296980,"statfs":{"total":96636764160,"available":96560107520,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574920,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92912,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94278928,"statfs":{"total":96636764160,"available":96541622272,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574921,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74860,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296980,"statfs":{"total":96636764160,"available":96560107520,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574921,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60972,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310868,"statfs":{"total":96636764160,"available":96574328832,"internally_reserved":0,"allocated":3510272,"data_stored":1746913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60976,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310864,"statfs":{"total":96636764160,"available":96574324736,"internally_reserved":0,"allocated":3514368,"data_stored":1746932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":95,"seq":408021893170,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39508,"kb_used_data":3468,"kb_used_omap":6,"kb_used_meta":36025,"kb_avail":94332332,"statfs":{"total":96636764160,"available":96596307968,"internally_reserved":0,"allocated":3551232,"data_stored":2337281,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":6429,"internal_metadata":36890339},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6985,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4834,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T00:39:16.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-24T00:39:17.157 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T00:39:17.157 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T00:39:17.157 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-24T00:39:17.157 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:17.244 INFO:tasks.ceph.osd.0:Stopped 2024-04-24T00:39:17.244 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-24T00:39:17.244 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:17.322 INFO:tasks.ceph.osd.1:Stopped 2024-04-24T00:39:17.322 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-24T00:39:17.322 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:17.398 INFO:tasks.ceph.osd.2:Stopped 2024-04-24T00:39:17.398 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-24T00:39:17.398 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:17.497 INFO:tasks.ceph.osd.3:Stopped 2024-04-24T00:39:17.497 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-24T00:39:17.497 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:22.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-24T00:39:23.599 INFO:tasks.ceph.osd.4:Stopped 2024-04-24T00:39:23.599 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-24T00:39:23.600 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:23.676 INFO:tasks.ceph.osd.5:Stopped 2024-04-24T00:39:23.677 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-24T00:39:23.677 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:23.754 INFO:tasks.ceph.osd.6:Stopped 2024-04-24T00:39:23.754 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-24T00:39:23.754 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:28.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-24T00:39:28.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-24T00:39:29.856 INFO:tasks.ceph.osd.7:Stopped 2024-04-24T00:39:29.856 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T00:39:29.857 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-24T00:39:29.857 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:29.885 INFO:tasks.ceph.mgr.y:Stopped 2024-04-24T00:39:29.885 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-24T00:39:29.885 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:29.916 INFO:tasks.ceph.mgr.x:Stopped 2024-04-24T00:39:29.916 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T00:39:29.916 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-24T00:39:29.916 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:29.950 INFO:tasks.ceph.mon.a:Stopped 2024-04-24T00:39:29.950 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-24T00:39:29.951 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:30.010 INFO:tasks.ceph.mon.c:Stopped 2024-04-24T00:39:30.011 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-24T00:39:30.011 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T00:39:30.035 INFO:tasks.ceph.mon.b:Stopped 2024-04-24T00:39:30.035 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T00:39:30.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-04-24T00:39:30.068 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:39:30.069 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-24T00:39:30.327 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:39:30.327 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-24T00:39:30.595 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:39:30.595 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-24T00:39:30.771 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:39:30.772 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-24T00:39:31.091 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:39:31.091 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-24T00:39:31.341 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:39:31.341 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-24T00:39:31.550 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:39:31.550 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-24T00:39:31.733 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:39:31.733 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-24T00:39:31.906 INFO:tasks.ceph:Archiving mon data... 2024-04-24T00:39:31.907 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/data/mon.a.tgz 2024-04-24T00:39:31.907 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-04-24T00:39:31.913 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.eNf1yobBe8 2024-04-24T00:39:31.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.eNf1yobBe8 2024-04-24T00:39:32.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.eNf1yobBe8 2024-04-24T00:39:32.152 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.eNf1yobBe8 is 540KB 2024-04-24T00:39:32.215 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.eNf1yobBe8 2024-04-24T00:39:32.219 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/data/mon.c.tgz 2024-04-24T00:39:32.219 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-04-24T00:39:32.263 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.3s0DT6us2l 2024-04-24T00:39:32.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.3s0DT6us2l 2024-04-24T00:39:32.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.3s0DT6us2l 2024-04-24T00:39:32.551 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.3s0DT6us2l is 720KB 2024-04-24T00:39:32.614 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.3s0DT6us2l 2024-04-24T00:39:32.619 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/data/mon.b.tgz 2024-04-24T00:39:32.620 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-04-24T00:39:32.626 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.L4pZLEF51A 2024-04-24T00:39:32.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.L4pZLEF51A 2024-04-24T00:39:32.847 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.L4pZLEF51A 2024-04-24T00:39:32.911 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.L4pZLEF51A is 720KB 2024-04-24T00:39:32.977 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.L4pZLEF51A 2024-04-24T00:39:32.983 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T00:39:32.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T00:39:32.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T00:39:33.046 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T00:39:33.088 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T00:39:33.088 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T00:39:33.089 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/remote/smithi064/crash 2024-04-24T00:39:33.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T00:39:33.104 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/remote/smithi167/crash 2024-04-24T00:39:33.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T00:39:33.143 INFO:tasks.ceph:Compressing logs... 2024-04-24T00:39:33.144 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T00:39:33.150 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T00:39:33.171 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30761.log 2024-04-24T00:39:33.172 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45048.log 2024-04-24T00:39:33.172 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36344.log 2024-04-24T00:39:33.172 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30761.log.gz 2024-04-24T00:39:33.173 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45048.log.gz 2024-04-24T00:39:33.173 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43904.log 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41804.log 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36344.log.gz 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40698.log 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43904.log.gz 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41804.log.gz 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41299.log 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36508.log 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40698.log.gz 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30640.log 2024-04-24T00:39:33.190 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41299.log.gz 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36508.log.gz 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44996.log 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43500.log 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30640.log.gz 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28871.log 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44996.log.gz 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33772.log 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43500.log.gz 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44652.log 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28871.log.gz 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40182.log 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33772.log.gz 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44652.log.gz 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38970.log 2024-04-24T00:39:33.191 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45386.log 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40182.log.gz 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43105.log 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38970.log.gz 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45386.log.gz 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35631.log 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41737.log 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43105.log.gz 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44558.log 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35631.log.gz 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41737.log.gz 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33800.log 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39801.log 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44558.log.gz 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40881.log 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33800.log.gz 2024-04-24T00:39:33.192 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31715.log 0.0% 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39801.log.gz 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42496.log 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40881.log.gz 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31715.log.gz 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42496.log.gz 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43324.log 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33565.log 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43324.log.gzgzip 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43748.log 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36999.log 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33565.log.gz 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43748.log.gz 2024-04-24T00:39:33.193 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44892.log 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32537.log 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36999.log.gz 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44892.log.gz 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41158.log 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30038.log 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32537.log.gz 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41158.log.gz 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38098.log 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29952.log 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30038.log.gz 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38098.log.gz 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32298.log 2024-04-24T00:39:33.194 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39657.log 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29952.log.gz 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32298.log.gz 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35520.log 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28794.log 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39657.log.gz 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35520.log.gz 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40771.log 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28794.log.gz 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40771.log.gz 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44866.log 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36422.log 2024-04-24T00:39:33.195 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.44866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44866.log.gz 2024-04-24T00:39:33.196 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35981.log 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25084.log 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25132.log 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.25084.log: /var/log/ceph/ceph-client.admin.25132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25132.log.gz 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25084.log.gz 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25163.log 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.25163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25163.log.gz 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25012.log 2024-04-24T00:39:33.233 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25108.log 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.25012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25012.log.gz 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.25108.log: /var/log/ceph/ceph-mgr.x.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25108.log.gz 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25060.log 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.25060.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25187.log 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25060.log.gz 2024-04-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-24T00:39:33.235 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25036.log 2024-04-24T00:39:33.235 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.25187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25187.log.gz 2024-04-24T00:39:33.235 INFO:teuthology.orchestra.run.smithi167.stderr: 85.8% -- replaced with /var/log/ceph/ceph.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-24T00:39:33.235 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-24T00:39:33.235 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.25036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25036.log.gz 2024-04-24T00:39:33.235 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.4.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T00:39:33.259 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35382.log 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36422.log.gz 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35981.log.gz 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30111.log 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38995.log 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35382.log.gz 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43956.log 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30111.log.gz 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38995.log.gz 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40747.log 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32359.log 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43956.log.gz 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31586.log 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40747.log.gz 2024-04-24T00:39:33.260 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32359.log.gz 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38038.log 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28845.log 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31586.log: /var/log/ceph/ceph-client.admin.38038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31586.log.gz 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38038.log.gz 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44349.log 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30860.log 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28845.log.gz 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44349.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44349.log.gz 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38887.log 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.30860.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30187.log 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30860.log.gz 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30996.log 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38887.log.gz 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30187.log.gz 2024-04-24T00:39:33.261 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30787.log 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29180.log 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30996.log.gz 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30260.log 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30787.log.gz 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29180.log.gz 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36084.log 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39152.log 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30260.log.gz 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36084.log.gz 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38336.log 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38034.log 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39152.log.gz 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38336.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38336.log.gz 2024-04-24T00:39:33.262 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42601.log 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29166.log 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38034.log.gz 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44159.log 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42601.log.gz 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29166.log.gz 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44165.log 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44159.log.gz 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39892.log 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36693.log 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44165.log.gz 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42337.log 2024-04-24T00:39:33.263 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39892.log.gz 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29759.log 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36693.log.gz 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34946.log 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42337.log.gz 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37727.log 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29759.log.gz 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.34946.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42099.log 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34946.log.gz 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29185.log 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37727.log.gz 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42099.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42099.log.gz 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41621.log 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29185.log.gz 2024-04-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44166.log 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44788.log 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41621.log.gz 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29677.log 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44166.log.gz 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44590.log 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44788.log.gz 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37181.log 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29677.log.gz 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40152.log 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44590.log.gz 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32329.log 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37181.log.gz 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44814.log 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40152.log.gz 2024-04-24T00:39:33.265 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32329.log.gz 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41923.log 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45074.log 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44814.log.gz 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37878.log 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41923.log.gz 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45074.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45074.log.gz 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37878.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.37878.log.gz /var/log/ceph/ceph-client.admin.36136.log 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40337.log 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43501.log 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36136.log.gz 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44363.log 2024-04-24T00:39:33.266 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40337.log.gz 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31430.log 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43501.log.gz 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32960.log 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44363.log.gz 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31857.log 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31430.log.gz 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38813.log 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32960.log.gz 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30162.log 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31857.log.gz 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31347.log 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38813.log.gz 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30162.log.gz 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32354.log 2024-04-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31347.log.gz 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44163.log 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40922.log 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32354.log.gz 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45334.log 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44163.log.gz 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30951.log 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40922.log.gz 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29805.log 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45334.log.gz 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45230.log 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30951.log.gz 2024-04-24T00:39:33.268 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38485.log 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29805.log.gz 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42234.log 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45230.log.gz 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28977.log 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38485.log.gz 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43155.log 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42234.log.gz 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40122.log 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28977.log.gz 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37269.log 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43155.log.gz 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35477.log 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40122.log.gz 2024-04-24T00:39:33.269 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39081.log 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37269.log.gz 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42288.log 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35477.log.gz 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35154.log 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39081.log.gz 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44762.log 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42288.log.gz 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33373.log 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35154.log.gz 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43930.log 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44762.log.gz 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36057.log 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33373.log.gz 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43252.log 2024-04-24T00:39:33.270 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43930.log.gz 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29177.log 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36057.log.gz 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39437.log 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43252.log.gz 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42725.log 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29177.log.gz 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35824.log 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39437.log.gz 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37644.log/var/log/ceph/ceph-client.admin.42725.log: 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42725.log.gz 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30516.log 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35824.log.gz 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35014.log 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37644.log.gz 2024-04-24T00:39:33.271 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42680.log 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30516.log.gz 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33857.log 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35014.log.gz 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35745.log 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42680.log.gz 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39724.log 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33857.log.gz 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45360.log 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35745.log.gz 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29387.log 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39724.log.gz 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43403.log 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45360.log.gz 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37964.log 2024-04-24T00:39:33.272 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29387.log.gz 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44364.log 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43403.log.gz 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30306.log 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37964.log.gz 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37033.log 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44364.log.gz 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31171.log 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30306.log.gz 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28923.log 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37033.log.gz 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41025.log 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31171.log.gz 2024-04-24T00:39:33.273 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28923.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28923.log.gz 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44034.log 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41025.log.gz 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40427.log 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42391.log 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44034.log.gz 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38632.log/var/log/ceph/ceph-client.admin.40427.log: 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40427.log.gz 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42391.log.gz 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42261.log 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38632.log.gz 2024-04-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.42261.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30893.log 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42261.log.gz 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34765.log 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr: 26.1% -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37926.log 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30893.log.gz 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32445.log 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34765.log.gz 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43560.log 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37926.log.gz 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36030.log 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32445.log.gz 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37224.log 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43560.log.gz 2024-04-24T00:39:33.275 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36030.log.gz 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25177.log 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37224.log.gz 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36744.log 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr: 29.1% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gz 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.25177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29651.log 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25177.log.gz 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29181.log 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36744.log.gz 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44458.log 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29651.log.gz 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29181.log.gz 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33688.log 2024-04-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44458.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42605.log 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43256.log 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33688.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33621.log 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42605.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39436.log 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43256.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29802.log 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33621.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.39436.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43052.log 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29802.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30449.log 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30605.log 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43052.log.gz 2024-04-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37503.log 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30449.log.gz 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43184.log 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30605.log.gz 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44840.log 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37503.log.gz 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39999.log 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43184.log.gz 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44344.log 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44840.log.gz 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38597.log 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39999.log.gz 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45282.log 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44344.log.gz 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37451.log 2024-04-24T00:39:33.278 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38597.log.gz 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32261.log 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45282.log.gz 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35933.log 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37451.log.gz 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43639.log 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32261.log.gz 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.35933.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38513.log 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35933.log.gz 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31471.log 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43639.log.gz 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37299.log 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38513.log.gz 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38155.log 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31471.log.gz 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44153.log 2024-04-24T00:39:33.279 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37299.log.gz 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38155.log.gz 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44153.log.gz 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41564.log 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41484.log 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29157.log 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41564.log.gz 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20244.log 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41484.log.gz 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29157.log.gz 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31748.log 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.tmp-client.admin.20244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28975.log 2024-04-24T00:39:33.280 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20244.log.gz 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38242.log 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31748.log.gz 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36177.log 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28975.log.gz 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38242.log.gz 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44060.log 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37127.log 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36177.log.gz 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32166.log 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44060.log.gz 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37127.log.gz 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34028.log 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28819.log 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32166.log.gz 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32503.log 2024-04-24T00:39:33.281 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34028.log.gz 2024-04-24T00:39:33.282 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38216.log 2024-04-24T00:39:33.282 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28819.log.gz 2024-04-24T00:39:33.282 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31103.log 2024-04-24T00:39:33.282 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32503.log: /var/log/ceph/ceph-client.admin.38216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32503.log.gz 2024-04-24T00:39:33.282 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38216.log.gz 2024-04-24T00:39:33.282 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32203.log 2024-04-24T00:39:33.282 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42977.log 2024-04-24T00:39:33.282 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31103.log.gz 2024-04-24T00:39:33.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-24T00:39:33.323 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32203.log.gz 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25022.log 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42977.log.gz 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36251.log 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40216.log: /var/log/ceph/ceph-client.admin.25022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25022.log.gz 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30473.log 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36458.log 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36251.log.gz 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29993.log 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30473.log.gz 2024-04-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31533.log 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36458.log.gz 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29599.log 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29993.log: /var/log/ceph/ceph-client.admin.31533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31533.log.gz 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32087.log 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34065.log 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29599.log.gz 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43774.log 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32087.log.gz 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34065.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30698.log 2024-04-24T00:39:33.325 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34065.log.gz 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29386.log 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43774.log.gz 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30423.log 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30698.log.gz 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44162.log 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29386.log: /var/log/ceph/ceph-client.admin.30423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41815.log 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29386.log.gz 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30423.log.gz 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44162.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38731.log 2024-04-24T00:39:33.326 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44162.log.gz 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41480.log 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41815.log.gz 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38731.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.42061.log 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38731.log.gz 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.41480.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44352.log 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41480.log.gz 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35897.log 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42061.log.gz 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37409.log 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44352.log.gz 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.35897.log: -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35897.log.gz 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37536.log 2024-04-24T00:39:33.327 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37409.log.gz 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43878.log 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42791.log 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37536.log.gz 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43878.log.gz 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35407.log 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42791.log.gz 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29573.log 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35407.log.gz 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43826.log 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32121.log 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29573.log.gz 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43826.log.gz 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43449.log 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29368.log 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32121.log.gz 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31062.log 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43449.log.gz 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37780.log 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29368.log.gz 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36625.log 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31062.log: /var/log/ceph/ceph-client.admin.37780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28961.log 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31062.log.gz 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37780.log.gz 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36625.log.gz 2024-04-24T00:39:33.329 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44918.log 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31819.log 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44918.log: /var/log/ceph/ceph-client.admin.28961.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44918.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.35042.log 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28961.log.gz 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31819.log.gz 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35093.log 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45204.log 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35042.log: /var/log/ceph/ceph-client.admin.35093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39542.log 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35042.log.gz 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35093.log.gz 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45204.log.gz 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38558.log 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43320.log 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39542.log.gz 2024-04-24T00:39:33.330 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41117.log 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38558.log.gz 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43320.log.gz 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37685.log 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29625.log 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41117.log.gz 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40980.log 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37685.log.gz 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.29625.log: -- /var/log/ceph/ceph-client.admin.31794.log 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29625.log.gz 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36663.log 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40980.log.gz 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37555.log 2024-04-24T00:39:33.331 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31794.log.gz 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.36663.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28976.log 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36663.log.gz 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28970.log 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37555.log.gz 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28976.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 0.0% 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28976.log.gz 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37375.log 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28970.log.gz 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42470.log 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44086.log 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37375.log.gz 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42470.log.gz 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35564.log 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-04-24T00:39:33.332 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44086.log.gz 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35564.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43670.log 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25281.log 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33522.log 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43670.log.gz 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.25281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25281.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38788.log 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33522.log.gz 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40040.log 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38398.log 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38788.log: /var/log/ceph/ceph-client.admin.40040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38788.log.gz 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40040.log.gz 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31638.log 2024-04-24T00:39:33.333 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39360.log 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38398.log: /var/log/ceph/ceph-client.admin.31638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38398.log.gz 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31638.log.gz 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37847.log 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28897.log 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37847.log: /var/log/ceph/ceph-client.admin.39360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39360.log.gz 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37847.log.gz 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45100.log 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29703.log 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28897.log: /var/log/ceph/ceph-client.admin.45100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28897.log.gz 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45100.log.gz 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41447.log 2024-04-24T00:39:33.334 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44378.log 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29703.log: /var/log/ceph/ceph-client.admin.41447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29703.log.gz 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41447.log.gz 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43613.log 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43190.log 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44378.log.gz 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43613.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36927.log 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29729.log 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43190.log.gz 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36927.log.gz 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44362.log 2024-04-24T00:39:33.335 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30311.log 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29729.log.gz 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28718.log 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44362.log.gz 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30311.log.gz 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31335.log 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41704.log 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28718.log: /var/log/ceph/ceph-client.admin.31335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28718.log.gz 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31335.log.gz 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41663.log 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39112.log 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41704.log.gz 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41663.log.gz 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38677.log 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39112.log.gz 2024-04-24T00:39:33.336 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35925.log 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36786.log 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38677.log: /var/log/ceph/ceph-client.admin.35925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38677.log.gz 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35925.log.gz 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30230.log 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36786.log: /var/log/ceph/ceph.audit.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36786.log.gz 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37814.log 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30230.log.gz 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43722.log 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37814.log.gz 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42016.log 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43722.log.gz 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-04-24T00:39:33.337 INFO:teuthology.orchestra.run.smithi064.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42016.log.gz 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41188.log 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44112.log 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40493.log: /var/log/ceph/ceph-client.admin.41188.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41188.log.gz 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44566.log 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44112.log: /var/log/ceph/ceph-client.admin.37477.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44112.log.gz 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33978.log 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33425.log 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44566.log: /var/log/ceph/ceph-client.admin.33978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44566.log.gz 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33978.log.gz 2024-04-24T00:39:33.338 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43445.log 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43587.log 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33425.log.gz 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43445.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43587.log.gz 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33561.log 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41971.log 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41377.log: /var/log/ceph/ceph-client.admin.33561.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33561.log.gz 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42862.log 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29800.log 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41971.log.gz 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42862.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40307.log 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29800.log.gz 2024-04-24T00:39:33.339 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41257.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35568.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40307.log.gz 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41257.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45410.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41257.log.gz 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35568.log.gz 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.32577.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33717.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45410.log.gz 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.32577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32014.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33717.log.gz 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44944.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32014.log.gz 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45022.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44944.log.gz 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44554.log 2024-04-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45022.log.gz 2024-04-24T00:39:33.348 INFO:teuthology.orchestra.run.smithi167.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-24T00:39:33.367 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35348.log 2024-04-24T00:39:33.367 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44554.log.gz 2024-04-24T00:39:33.367 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36317.log 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35348.log.gz 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29393.log 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36317.log: 0.0% 95.2% -- replaced with /var/log/ceph/objectstore_tool.32577.log.gz 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36317.log.gz 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33421.log 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42521.log 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33421.log: /var/log/ceph/ceph-client.admin.29393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29393.log.gz 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33421.log.gz 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35798.log 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37087.log 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42521.log: /var/log/ceph/ceph-client.admin.35798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42521.log.gz 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35798.log.gz 2024-04-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31611.log 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36512.log 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37087.log: /var/log/ceph/ceph-client.admin.31611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37087.log.gz 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31611.log.gz 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44619.log 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30115.log 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36512.log: /var/log/ceph/ceph-client.admin.44619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36512.log.gz 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44619.log.gz 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45308.log 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44970.log 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30115.log: /var/log/ceph/ceph-client.admin.45308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30115.log.gz 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45308.log.gz 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45178.log 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43982.log 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44970.log: /var/log/ceph/ceph-client.admin.45178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44970.log.gz 2024-04-24T00:39:33.369 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45178.log.gz 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36110.log 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38402.log 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43982.log: /var/log/ceph/ceph-client.admin.36110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43982.log.gz 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36110.log.gz 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30571.log 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33896.log 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38402.log: /var/log/ceph/ceph-client.admin.30571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38402.log.gz 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30571.log.gz 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44164.log 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43001.log 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33896.log.gz 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44164.log.gz 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41897.log 2024-04-24T00:39:33.370 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43001.log.gz 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28965.log 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40589.log 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41897.log.gz 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28965.log.gz 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45152.log 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40589.log.gz 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35241.log 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29389.log 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45152.log.gz 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35241.log.gz 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31107.log 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44565.log 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29389.log: /var/log/ceph/ceph-client.admin.31107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29389.log.gz 2024-04-24T00:39:33.371 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31107.log.gz 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40528.log 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44565.log.gz 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40528.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43696.log 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36518.log 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40650.log 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43696.log.gz 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36518.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36836.log 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40650.log.gz 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.29792.log 2024-04-24T00:39:33.372 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31244.log 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36836.log.gz 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.0.29792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44156.log 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31244.log.gz 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr: 95.4% -- replaced with /var/log/ceph/ceph-client.0.29792.log.gz 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44156.log.gz 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29178.log 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30314.log 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29178.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39334.log 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30314.log.gz 2024-04-24T00:39:33.373 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28973.log 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29388.log 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39334.log: /var/log/ceph/ceph-client.admin.28973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39334.log.gz 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28973.log.gz 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40654.log 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29388.log.gz 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29369.log 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40654.log.gz 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43852.log 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30520.log 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29369.log: /var/log/ceph/ceph-client.admin.43852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29369.log.gz 2024-04-24T00:39:33.374 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43852.log.gz 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40211.log 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45256.log 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30520.log.gz 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40211.log.gz 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31976.log 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45256.log.gz 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38295.log 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40092.log 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31976.log: /var/log/ceph/ceph-client.admin.38295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31976.log.gz 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38295.log.gz 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31269.log 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35661.log 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40092.log: /var/log/ceph/ceph-client.admin.31269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40092.log.gz 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31269.log.gz 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29379.log 2024-04-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42887.log 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35661.log.gz 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29379.log.gz 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28966.log 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42887.log.gz 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36901.log 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39230.log 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.28966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28966.log.gz 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36901.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39915.log 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39230.log.gz 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41340.log 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39546.log 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39915.log: /var/log/ceph/ceph-client.admin.41340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39915.log.gz 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41340.log.gz 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42152.log 2024-04-24T00:39:33.376 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42153.log 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39546.log.gz 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42152.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44008.log 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42153.log.gz 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31935.log 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29179.log 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44008.log: /var/log/ceph/ceph-client.admin.31935.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44008.log.gz 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31935.log.gz 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41927.log 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36292.log 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29179.log: /var/log/ceph/ceph-client.admin.41927.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29179.log.gz 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41927.log.gz 2024-04-24T00:39:33.377 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35287.log 2024-04-24T00:39:33.381 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36719.log 2024-04-24T00:39:33.381 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35287.log: /var/log/ceph/ceph-client.admin.36292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36292.log.gz 2024-04-24T00:39:33.381 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35287.log.gz 2024-04-24T00:39:33.381 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43800.log 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35657.log 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36719.log.gz 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43800.log.gzgzip 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39838.log 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35657.log.gz 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44615.log 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45126.log 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39838.log: /var/log/ceph/ceph-client.admin.44615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39838.log.gz 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44615.log.gz 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40393.log 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30674.log 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45126.log.gz 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40393.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42208.log 2024-04-24T00:39:33.382 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:33.383 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30674.log.gz 2024-04-24T00:39:33.383 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42836.log 2024-04-24T00:39:33.383 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42208.log: 0.0%/var/log/ceph/ceph-client.admin.42836.log: -- replaced with /var/log/ceph/ceph-client.admin.42208.log.gz 2024-04-24T00:39:33.383 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42836.log.gz 2024-04-24T00:39:33.505 INFO:teuthology.orchestra.run.smithi167.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-24T00:39:33.549 INFO:teuthology.orchestra.run.smithi064.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-24T00:39:33.651 INFO:teuthology.orchestra.run.smithi064.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-24T00:39:35.190 INFO:teuthology.orchestra.run.smithi064.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-24T00:39:35.380 INFO:teuthology.orchestra.run.smithi167.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-24T00:39:35.470 INFO:teuthology.orchestra.run.smithi064.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-24T00:39:35.531 INFO:teuthology.orchestra.run.smithi064.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-24T00:39:35.790 INFO:teuthology.orchestra.run.smithi167.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-24T00:39:36.045 INFO:teuthology.orchestra.run.smithi064.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-24T00:39:36.046 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-24T00:39:36.046 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m2.894s 2024-04-24T00:39:36.046 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m10.022s 2024-04-24T00:39:36.046 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.484s 2024-04-24T00:39:38.583 INFO:teuthology.orchestra.run.smithi167.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-24T00:39:39.310 INFO:teuthology.orchestra.run.smithi167.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-24T00:39:39.311 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-04-24T00:39:39.311 INFO:teuthology.orchestra.run.smithi167.stderr:real 0m6.111s 2024-04-24T00:39:39.311 INFO:teuthology.orchestra.run.smithi167.stderr:user 0m15.991s 2024-04-24T00:39:39.311 INFO:teuthology.orchestra.run.smithi167.stderr:sys 0m0.635s 2024-04-24T00:39:39.312 INFO:tasks.ceph:Archiving logs... 2024-04-24T00:39:39.312 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/remote/smithi064/log 2024-04-24T00:39:39.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T00:39:40.680 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/remote/smithi167/log 2024-04-24T00:39:40.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T00:39:42.606 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T00:39:42.615 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T00:39:42.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T00:39:42.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T00:39:42.697 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T00:39:42.707 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T00:39:42.707 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T00:39:42.709 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T00:39:42.720 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T00:39:42.720 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2024-04-24T00:39:42.720 INFO:teuthology.orchestra.run.smithi064.stdout:*hv01.front.sepi 44.4.53.2 2 u 9 64 377 0.047 -3.478 1.242 2024-04-24T00:39:42.721 INFO:teuthology.orchestra.run.smithi064.stdout:+hv02.front.sepi 63.231.80.2 3 u 11 64 377 0.065 +0.169 0.709 2024-04-24T00:39:42.721 INFO:teuthology.orchestra.run.smithi064.stdout:+hv03.front.sepi 74.6.168.72 3 u 16 64 377 0.088 -0.696 0.641 2024-04-24T00:39:42.721 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:39:42.722 INFO:teuthology.orchestra.run.smithi167.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T00:39:42.722 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================== 2024-04-24T00:39:42.722 INFO:teuthology.orchestra.run.smithi167.stdout:+hv01.front.sepi 44.4.53.2 2 u 10 64 377 0.079 -3.783 1.931 2024-04-24T00:39:42.722 INFO:teuthology.orchestra.run.smithi167.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 64 377 0.102 +0.468 1.596 2024-04-24T00:39:42.722 INFO:teuthology.orchestra.run.smithi167.stdout:*hv03.front.sepi 74.6.168.72 3 u 7 64 377 0.076 -0.770 1.343 2024-04-24T00:39:42.722 INFO:teuthology.orchestra.run.smithi167.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T00:39:42.723 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T00:39:42.746 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T00:39:42.746 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T00:39:42.755 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T00:39:42.763 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T00:39:42.772 INFO:teuthology.task.internal:Duration was 970.605002 seconds 2024-04-24T00:39:42.773 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T00:39:42.860 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T00:39:42.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T00:39:42.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T00:39:42.901 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T00:39:42.901 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-04-24T00:39:42.901 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T00:39:42.950 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-04-24T00:39:42.950 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T00:39:42.961 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T00:39:42.961 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T00:39:42.993 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T00:39:43.031 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T00:39:43.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T00:39:43.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T00:39:43.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T00:39:43.159 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T00:39:43.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T00:39:43.172 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T00:39:43.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T00:39:43.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T00:39:43.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T00:39:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-04-24T00:39:43.244 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-04-24T00:39:43.278 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T00:39:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:39:43.304 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T00:39:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T00:39:43.323 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T00:39:43.333 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T00:39:43.334 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/remote/smithi064 2024-04-24T00:39:43.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T00:39:43.372 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-23_21:16:02-rbd-squid-distro-default-smithi/7670533/remote/smithi167 2024-04-24T00:39:43.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T00:39:43.392 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T00:39:43.393 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T00:39:43.409 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T00:39:43.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T00:39:43.453 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T00:39:43.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T00:39:43.495 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T00:39:43.495 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T00:39:43.498 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T00:39:43.501 INFO:teuthology.orchestra.run.smithi064.stdout: 658634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 00:39 /home/ubuntu/cephtest 2024-04-24T00:39:43.503 INFO:teuthology.orchestra.run.smithi167.stdout: 658638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 00:39 /home/ubuntu/cephtest 2024-04-24T00:39:43.504 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T00:39:43.514 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T00:39:43.537 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_deep_copy} duration: 970.6050024032593 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-24T00:39:43.537 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T00:39:43.633 INFO:teuthology.run:pass